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Overview 


The nation’s community colleges play a central role in producing a more educated workforce and 
promoting social mobility. They serve about 40 percent of all college students and, not surprisingly, 
they serve a disproportionate number of low-income and underrepresented students. But most students 
who enter these colleges do not graduate — only about a third of entering students earn a degree or 
certificate within six years. 


Among the many programs that have attempted to increase graduation rates, one program stands out. 
Developed by the City University of New York (CUNY), the Accelerated Study in Associate Pro- 
grams (ASAP) is a comprehensive program that provides students with up to three years of financial 
and academic support and other support services. Along with those services and other forms of support 
comes an obligation to attend full time and participate in essential program services. An experimental 
evaluation of CUNY ASAP found that the program nearly doubled graduation rates after three years. 


This report presents findings through three years from a replication of the ASAP model at three com- 
munity colleges in Ohio. Low-income students were randomly assigned either to a program group, 
who could participate in their colleges’ new programs based closely on ASAP (called the Ohio Pro- 
grams), or to a control group, who could receive the usual college services. Comparing the two groups’ 
outcomes provides an estimate of the Ohio Programs’ effects. 


Findings 
e The Ohio Programs were generally managed and staffed as planned and implemented as designed. 


e The programs created a noticeable contrast between the experiences of program and control group 
students in all areas where the model had components. 


e The Ohio Programs nearly doubled degree receipt through three years and led to an increase in 
transfers to four-year colleges. 


e The Ohio Programs had positive effects on enrollment, full-time enrollment, and credits earned. 


e The Ohio programs had positive effects for various types of students, including those who entered 
with and without developmental (remedial) education requirements. 


e The Ohio Programs cost more per student than usual services, but also led to a lower cost per 
degree. 


The findings from the evaluation add to a body of evidence showing that comprehensive programs 
that offer academic, financial, and other forms of student support for multiple years are an effective 
way to increase college completion rates. The fact that the earlier CUNY ASAP findings have now 
been replicated in Ohio suggests that ASAP in particular can serve as a national model to help students 
succeed. 
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Preface 


Community colleges provide a relatively affordable college option for millions of students 
across the country, many of whom come from low-income families and many of whom enter 
with some need for remedial instruction before starting college-level work. Completion rates 
are low for community college students in general, but especially so for such students. Many 
reforms have been tried to help them succeed, but few have substantially boosted college 
completion. 


In 2007, the City University of New York (CUNY), with the support of the New York 
City Center for Economic Opportunity, launched the Accelerated Study in Associate Programs 
(ASAP) to encourage and support community college students to attend school full time and 
graduate. The program provides financial assistance, special courses, enhanced advising, and 
other support services for three full years. The findings from CUNY ASAP were very encourag- 
ing and the largest to date found from any program MDRC has evaluated at community colleges: 
The program nearly doubled graduation rates after three years, from 22 percent to 40 percent. 


Can ASAP serve as a national model for helping more students get through college? The 
findings from CUNY, although striking, are just a starting point. It has become increasingly clear, 
as the evaluation field has grown and evolved, that findings from one evaluation are not enough 
to make policy. Programs that might work in one context often do not work in another. Even 
when tested in the same context, positive effects found in one trial are often not found in a subse- 
quent trial. For that reason, it was critical to test the ASAP model at different colleges and for 
different types of students. 


The ASAP Ohio demonstration provides that replication, and the results are encouraging. 
In fact, the results are remarkably similar to those from CUNY. The Ohio Programs (as the pro- 
grams at Ohio colleges modeled on ASAP are called here) also nearly doubled graduation rates 
after three years and increased transfers to four-year colleges by about the same amount as CUNY 
ASAP. Positive effects were found at all three participating colleges and for all subgroups of 
students examined. Although the program cost more per student than the usual services, it led to 
a lower cost per degree. 


CUNY is continuing to expand its program across its own system and beyond New York 

City, providing guidance to other colleges in New York State and California. MDRC is evaluating 

one of those efforts to provide more evidence on ASAP. But the findings to date strongly suggest 
that the ASAP model is an investment worth considering for community colleges nationwide. 

Virginia Knox 

President, MDRC 
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Executive Summary 


The nation’s community colleges play a central role in producing a more educated workforce and 
promoting social mobility. They serve about 40 percent of all college students and, not surpris- 
ingly, they serve a disproportionate number of low-income and underrepresented students. How- 
ever, although enrollment has generally increased over time, completion rates have not. Among 
first-time students entering public two-year colleges, only about a third earn degrees or certificates 
within six years.' Most also do not eventually transfer to four-year institutions. 


Many colleges have started providing additional services and support to students to help 
them stay in college and complete degrees. This report provides evidence on one such program, 
tested at three community colleges in Ohio. Developed by the City University of New York 
(CUNY), the Accelerated Study in Associate Programs (ASAP) provides students with up to 
three years of financial and academic support and other support services to address multiple bar- 
riers to student success, with the goal of helping more students graduate within three years. ASAP 
is both an opportunity and an obligation, in that students are required to attend full time and to 
participate in program services. 


An evaluation of CUNY ASAP found that it led to a large increase in graduation rates, 
with some of the largest effects observed among programs that have been evaluated.2 ASAP’s 
success led to questions about whether it could be a model for community colleges across the 
nation. To answer those questions, it would be important to assess whether it worked as well in 
another context. MDRC therefore joined with CUNY and the Ohio Department of Higher Edu- 
cation (ODHE) to implement and test the ASAP model at three community colleges in the state. 
ODHE and the colleges were seeking ways to address low completion rates among community 
college students and appreciated ASAP’s comprehensive nature and its idea of reciprocal obliga- 
tion. CUNY provided in-depth technical assistance to the colleges on the program model and how 
different components could be implemented, while ODHE coordinated knowledge sharing 
among college leaders and program staff members at the three colleges. MDRC provided opera- 
tional support, led the evaluation, and oversaw the demonstration. The project was launched in 
2014 and the colleges’ programs (referred to as the Ohio Programs) began operating in 2015. 
Eligible students who agreed to take part in the study were assigned at random to either a program 
group, eligible for the Ohio Programs’ services and support, or a control group, not eligible to 
participate in the Ohio Programs but eligible for the colleges’ standard services. 


An earlier report, documenting effects through two years, found that the colleges suc- 
cessfully implemented the Ohio Programs and that, compared with standard courses and services 


‘Alexandria Walton Radford, Lutz Berkner, Sara C. Wheeless, and Bryan Shepherd, Persistence and At- 
tainment of 2003-04 Beginning Postsecondary Students: After 6 Years. First Look, NCES 2011-151 (Washing- 
ton, DC: National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education, 
2010). 

Susan Scrivener, Michael J. Weiss, Alyssa Ratledge, Timothy Rudd, Colleen Sommo, and Hannah 
Fresques, Doubling Graduation Rates: Three-Year Effects of CUNY’s Accelerated Study in Associate Programs 
(ASAP) for Developmental Education Students (New York: MDRC, 2015). 


at the colleges, the programs led to large increases in enrollment and credit accumulation in each 
of the four semesters and more than doubled two-year graduation rates. 


This report presents findings on academic outcomes through three years, to assess 
whether these large effects persisted and whether they continue to match the magnitude of the 
effects found in New York. The report also assesses the implementation of the programs at the 
three colleges and presents a cost-effectiveness analysis, assessing the outcomes achieved per 
dollar spent. In sum, the programs were largely implemented as planned over the three-year pe- 
riod, with some exceptions. Students received the messages about full-time enrollment and on- 
time completion, and program group students participated in advising, tutoring, and career ser- 
vices at higher rates than their control group counterparts. The programs also continued to have 
large, positive effects on enrollment, school progress, and degree receipt. In addition, new effects 
emerged in Year 3 on transfers to four-year colleges. The findings are strikingly similar to those 
found for the original CUNY ASAP program, suggesting that the model’s comprehensive support 
can be implemented and help students beyond these two states. 


The Ascendium Education Group provided anchor funding for the demonstration and 
evaluation, supplemented by grants from a consortium of other philanthropies, including Amold 
Ventures, the Bill & Melinda Gates Foundation, the ECMC Foundation, the Ford Foundation, the 
Greater Cincinnati Foundation, Haile U.S. Bank Foundation, KnowledgeWorks, the Kresge 
Foundation, and Lumina Foundation. 


The Ohio Programs 


The Ohio Programs model (based closely on CUNY ASAP) was designed to address multiple 
potential barriers to student success simultaneously and to address them over three full years. The 
model as originally planned included the following components: 


Student Support 


e Advising. The programs would offer comprehensive advising from an adviser 
with a small caseload (about 125 students). Students were required to visit their 
advisers twice per month in the first semester and as directed based on need 
thereafter. 


e Tutoring. The programs would require students to attend tutoring if they were 
taking developmental (remedial) courses, on academic probation, or identified 
as struggling by a faculty member or adviser. 


e Career services. The programs also would require students to meet with a 
campus career services staff member or participate in an approved career ser- 
vices event once per semester. 


Financial Support 


e Tuition waiver. The programs would provide a tuition waiver that covered 
any gap between financial aid and college tuition and fees. 


e Monthly incentive. The programs would offer a monthly incentive in the form 
of a $50 gas/grocery gift card, contingent on participation in program services. 


e Textbook voucher. The programs would provide a voucher to cover the costs 
of textbooks. 


Course Enrollment 


e Blocked courses and consolidated schedules. Seats would be held for pro- 
gram students in specific sections of courses during the first year. 


e First-year seminar. New students would be required to take a first-year sem- 
inar (or “success course’’), ideally with other program students, covering topics 
such as study skills and note-taking. 


Requirements and Messages 


e Full-time enrollment. The programs would require students to attend college 
full time during the fall and spring semesters and encourage students to enroll 
in classes in the summer. 


e Taking developmental courses early. The programs would encourage stu- 
dents to take developmental courses early in their time in college. 


e Graduation within three years. The programs would encourage students to 
graduate within three years. 


Findings 


e The Ohio programs were generally managed and staffed as planned and 
implemented as designed. 


The programs had directors and advisers who worked solely with program group students 
daily. For the most part, colleges did not employ program-specific tutors or career advisers; stu- 
dents received these services primarily from existing centers on campus. The colleges provided 
students in the program group with most of the intended program components and services, with 
the exception of blocked course schedules. That component was not provided as planned in part 
because students could have up to 24 credits when they entered the study; many of them had 
already completed various developmental or general education courses that typically would have 
been included in a blocked schedule.* Students received the financial support — tuition waivers, 


3Colleges often require students to take non-major-specific courses such as math or English. These courses 
are referred to as “general education” courses. 


textbook assistance, and monthly incentives in the form of gift cards — as planned. Students’ 
participation in advising was consistently high over the course of their years in the programs. 
Students’ participation in tutoring and career services was lower than their participation in advis- 
ing. Over time, however, the staff adjusted the requirements (for example, by allowing the use of 
online tutoring services) to increase participation. 


e The programs created a noticeable contrast between the experiences of 
program and control group students in all areas where the model had 
components. 


Program group students were more likely than control group students to be engaged in 
advising, tutoring, and career services. Program group students participated in these services at 
higher rates than their control group counterparts and reported more meetings with student ser- 
vices staff members overall. For example, 63 percent of program group students reported using 
tutoring services during the first year of the programs, compared with 45 percent of control group 
services, an increase of 19 percentage points. The increase in the use of career counseling was 
even larger, at 34 percentage points. Program group students also reported meeting with advisers 
many more times each semester (about 13 times on average) than control group students (about 
5 times on average). Finally, program group students were more likely than control group students 
to report hearing the three central program messages about the importance of enrolling full time, 
enrolling during the summer, and graduating in three years. Fewer than half of control group 
students reported hearing these messages. 


e Program group students were less reliant on student loans, personal sav- 
ings, or support from family and friends to pay for college. 


The additional financial aid that students received from the programs led to a number 
of effects. First, students were less likely to report relying on other sources of assistance — 
including parents, relatives, friends, and savings — to help pay for college. In addition, students 
in the program group expressed less concern about their ability to pay for college. As an exam- 
ple, nearly a third of control group students reported that they chose not to register for college 
full time because they could not afford it, compared with about 16 percent of students in the 
program group. Program group students were also less likely than control group students to 
report not buying textbooks because of the cost. Interestingly, the programs do not appear to 
have reduced students’ need to work while in school. About two-thirds of students in the pro- 
gram and control groups worked during their first year enrolled, and those who did so worked 
an average of 30 hours per week. 


e The Ohio Programs nearly doubled degree receipt through three years 
and led to an increase in transfers to four-year colleges. 


Effects on degree receipt emerged by the third semester of the follow-up period and in- 
creased in each subsequent semester. By the end of Year 3 (the sixth semester), 35 percent of the 
students in the program group had earned degrees, compared with 19 percent of students in the 


control group, an increase of 16 percentage points (see Figure ES.1). As expected, all of the pro- 
grams’ effects on degree receipt are for associate’s degrees. Few students earned certificates, and 
even fewer earned bachelor’s degrees or higher degrees during this period. 


Figure ES.1 


The Ohio Programs Increased Degree Receipt and 
Transfers to Four-Year Colleges 
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SOURCE: MDRC calculations using data from the National Student Clearinghouse and 
transcript data from the study colleges. 


The Ohio Programs also increased the number of students registered at four-year col- 
leges. By the fifth and sixth semesters, students in both the program and control groups began 
transferring to four-year colleges. But higher percentages of program group students made those 
transfers. By the sixth semester, for example, 18 percent of program group students were regis- 
tered at four-year colleges, compared with 12 percent of control group students. 


e The Ohio Programs had positive effects on enrollment, full-time enroll- 
ment, and credits earned. 


Enrollment rates for both research groups fell over time, illustrating the challenges to 
persistence in college faced by community college students. However, students in the program 
group were more likely to be enrolled and to be enrolled full time in each of the first five semesters 
(see Figure ES.2). In the third semester, for example, 68 percent of program group students were 
enrolled at one of the participating colleges compared with 58 percent of control group students, 
an increase of 10 percentage points. The effect on full-time enrollment was even larger, at 19 
percentage points. 
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Figure ES.2 


The Ohio Programs Boosted Enrollment 
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SOURCE: MDRC calculations using data from the National Student Clearinghouse and transcript data from the study colleges. 


NOTES: Enrollment is based on all available data and combines spring and summer enrollment. 

FT = full-time, defined as enrollment in 12 or more credits and based on data from the college of random assignment only. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 
percent. 

Estimates are adjusted by site, cohort, gender, race/ethnicity, age, parental status, marital status, weekly hours worked, dependence on parents for 50 
percent or more of financial support, whether a student is the first family member to attend college, whether a student earned a high school diploma, the number 
of developmental education requirements at the time of random assignment, and intended enrollment level. 


Students in the program group also earned more credits throughout the period, in part 
because higher percentages of them were enrolled but also probably in part due to the additional 
support they received while enrolled and the requirement to enroll full time. The Ohio Programs 
led to an increase of just over 2 credits earned in each of the first three semesters and 1.5 credits 
in the fourth semester. Like the effects on enrollment, the effects on credits earned diminish over 
time and are no longer statistically significant by the sixth semester. In total, students in the pro- 
gram group earned 8.5 more credits over the follow-up period than their counterparts in the con- 
trol group. To some extent, effects on both current enrollment and credits earned are expected to 
fall over time as students earn degrees and leave college. As noted, by the end of the sixth semes- 
ter, 35 percent of students in the program group had earned degrees. 


e The Ohio Programs had positive effects for various types of students, 
including those who entered with and without developmental education 
requirements. 


Effects were estimated for several subgroups of students. Of particular interest were ef- 
fects for subgroups of students with different levels of academic preparation, specifically students 
who did and did not have developmental education requirements when they entered the study. 
The program led to increases in degree receipt for students with developmental education require- 
ments that were about the same size as the increases for those without these requirements. The 
original CUNY ASAP study focused only on students with developmental requirements, so the 
finding here for students with developmental requirements replicates those results and shows that 
the program model worked in this new setting for students who were less prepared academically. 
But the findings for students without developmental requirements confirm that more academi- 
cally prepared students can also benefit from these types of services and support. 


Effects were also explored for several other subgroups of students, defined by college, 
gender, race, age, and whether a student had a high diploma or equivalent, etc. For all the sub- 
groups examined, the estimated effects on three-year graduation rates were large, positive, and 
statistically significant, demonstrating that the program is effective for a variety of students. 


e The Ohio Programs cost more per student than the usual services, but 
also led to a lower cost per degree. 


The direct cost of the programs was $5,521 per program group member over three years, 
or $1,840 per year. This estimate includes $2,369 for administration and staffing, $1,517 for stu- 
dent services, and $1,635 for financial support. In total, after adding in the costs of educating 
more students (since the programs increased enrollment and the number of college courses taken), 
the colleges invested $8,030 more per program group member than they did per control group 
member. However, that investment also led to a large increase in degree receipt. Thus, the cost 
per degree earned for program group students was 22 percent lower than the cost per degree 
eared for control group students. 


Conclusion 


The findings from the evaluation add to a body of evidence showing that comprehensive pro- 
grams that provide academic, financial, and personal support to students for multiple years are an 
effective way to increase college completion rates. Equally important, however, the findings add 
to the evidence on the effectiveness of the CUNY ASAP model. The effects of the Ohio Programs 
are strikingly similar to those found from the CUNY ASAP evaluation, providing more evidence 
that the model works for different types of students. As an example, more of the students in the 
Ohio study are “nontraditional” than was the case in New York, meaning that the students are 
older, have children, or work full time. There was some concern that the model, with its full-time 
attendance requirement, would not work for these types of students, yet it appears it does work. 
The findings also show that the model can work in a different context. For example, while CUNY 
is a highly centralized university system, Ohio’s community colleges are decentralized, operating 
independently and making almost all decisions on their own. Practitioners and policymakers have 
already recognized ASAP as a national model to increase educational attainment, so it is note- 
worthy that the model has now produced similar results with different students and in a different 
context. 


The findings also illustrate that the model can be adapted to fit the local context and still 
have large effects. The Ohio colleges put program components into operation somewhat differ- 
ently than CUNY did. In some cases, the differences were planned from the outset: For example, 
the Ohio Programs provided gift cards for gas and food where CUNY ASAP provided unlimited 
MetroCards for the New York City transit system. In other cases, the differences were the result 
of adaptations the colleges made as the programs were being implemented, such as not offering 
blocked scheduling and allowing students to meet the tutoring requirements in multiple ways. 
However, the general model — of integrated services, other forms of support, and requirements, 
combined with active monitoring and outreach — was still preserved. 


The Ohio Programs cost less than the original CUNY ASAP in New York. But it is 
worth noting that the original program’s costs have dropped substantially, as CUNY has mod- 
ified it and expanded it to serve more students within the CUNY system. The CUNY ASAP 
model is also expanding beyond Ohio and New York City, to colleges such as Westchester 
Community College in New York and the colleges in the San Mateo County Community Col- 
lege System in California. The model does require an investment by the colleges. However, the 
findings from Ohio and New York suggest that it is an investment that can help substantial 
numbers of community college students graduate and help colleges continue to play an im- 
portant role in promoting social mobility. 


Chapter 1 


Introduction 


The labor market has changed dramatically over the past several decades, with the result that 
postsecondary education is now almost a requirement to earn a decent wage. The changes in the 
labor market include rising international and domestic outsourcing, increased technological 
change, and automation; these changes have increased wages at the top of the distribution, for 
workers with higher levels of education, and pushed down wages at the bottom.' Average workers 
with only a high school education, for example, make less in real terms today than they did in the 
early 1970s.? 


These trends in wages are not likely to change anytime soon, given the pace of techno- 
logical change and automation and continued outsourcing.* The importance of a college degree 
will only grow, leaving less-educated workers further behind. A central policy question is how to 
help more workers obtain educational credentials and increase their skills. In this effort, commu- 
nity colleges play a vital role. 


The nation’s community colleges serve about 40 percent of all college students and, not 
surprisingly, serve a disproportionate number of low-income and underrepresented students.* 
They are relatively accessible, typically with open admission policies, and they are on average 
much less expensive than four-year colleges or two-year private colleges. For a large share of the 
population, they represent an important first step to higher education and upward mobility. 


However, although enrollment in community colleges has generally increased over 
time, completion rates have not. Among first-time students entering public two-year colleges, 
only 15 percent graduate with a degree or certificate within three years.> After six years, that 
number increases to about 35 percent, while 46 percent of students have not earned a degree 
and are no longer enrolled in college at all.° Most students also do not eventually transfer to 
four-year institutions. 


The low completion rates may not be very surprising, given the barriers many low- 
income community college students face. For many of them, financial aid is not enough to cover 
the full costs of attendance (including tuition, fees, and living expenses), requiring them to work 
while attending. Others are the first in their families to attend college and may have difficulty 
navigating the college environment. Some students, particularly older students, may be juggling 
work and family responsibilities in addition to school. And others may have low levels of basic 


'See, for example, Autor and Dorn (2013); Goldin and Katz (2009). 
Economic Policy Institute (2019). 

3Brown and Loprest (2018). 

4Ma and Baum (2016). 

STfill et al. (2016). 

Radford, Berkner, Wheeless, and Shepherd (2010). 


skills, which makes it difficult for them to complete college-level course work, and may also 
require them to complete remedial courses, referred to as “developmental education” courses, 
before they can enroll in college-credit courses. These students need additional help but their 
colleges are often not equipped to provide it, often for lack of funding. 


Many colleges have started providing additional assistance to address low completion 
rates. Examples include enhanced advising, financial incentives for meeting certain milestones or 
enrolling full time, changes in the disbursement of financial aid, and restructured developmental 
education classes.’ Evaluations of these policies suggest that they can increase student retention 
in college and degree completion, although the effects are sometimes mixed and generally modest 
in size. But one program stands out. 


Developed by the City University of New York (CUNY), the Accelerated Study in As- 
sociate Programs (ASAP) is a comprehensive program that provides students with up to three 
years of financial and academic support and other support services to address multiple barriers to 
student success, with the goal of helping more students graduate within three years. That support 
comes with an obligation, however, since students are required to attend full time and to partici- 
pate in program services. An experimental evaluation of CUNY ASAP found that the program 
nearly doubled graduation rates after three years.* After six years, students offered the program 
remained 10 percentage points more likely to have graduated than their counterparts who were 
not offered the program.’ Among postsecondary programs that have been evaluated using ran- 
domized controlled trials, these effects are some of the largest observed. '° 


The success of the program led to questions about whether it would work in other con- 
texts. If ASAP were to be a model for community colleges across the nation, then it would be 
important to assess whether it worked similarly well in another context, outside of New York City 
and for students other than those in CUNY community colleges. To answer this question, MDRC 
joined with CUNY and the Ohio Department of Higher Education (ODHE), which was attempt- 
ing to address the problem of low completion rates for community college students in Ohio, to 
develop the ASAP Ohio Demonstration. The notion of reciprocal obligation was one factor that 
resonated with officials in Ohio: Students eligible for Pell Grants would receive enhanced services 
and support contingent on meeting program requirements. 


The demonstration launched in 2014. Using ASAP as a model, three community colleges 
in Ohio began a new strategy (referred to as the Ohio Programs) to help more of their lower- 
performing students succeed. CUNY provided in-depth technical assistance, while ODHE coor- 
dinated knowledge sharing among college leaders and program staff members at the three 


"Bettinger and Baker (2014); Mayer, Patel, Rudd, and Ratledge (2015); Weissman, Cerna, and Cullinan 
(2019); Weiss and Headlam (2018). 

8Scrivener et al. (2015). CUNY also conducted its own quasi-experimental evaluation of the program with 
different groups of students. See Strumbos and Kolenovic (2017) and City University of New York (2019). 

°Weiss, Ratledge, Sommo, and Gupta (2019). 

'0See, for example, Bettinger and Baker (2014); Weiss, Visher, Weissman, and Wathington (2015); Mayer, 
Patel, Rudd, and Ratledge (2015); Goldrick-Rab, Harris, Kelchen, and Benson (2012); and Scrivener and Weiss 
(2009). 


colleges (referred to as the ASAP Ohio Network). MDRC provided operational support, led the 
evaluation, and oversaw the demonstration. 


An earlier report, documenting effects through two years, found that for the most part the 
colleges successfully implemented the programs based on the ASAP model and that compared 
with standard courses and services at the colleges, the Ohio Programs led to large increases in 
enrollment and credit accumulation in each of the four semesters after the study began, and more 
than doubled two-year graduation rates. This report presents findings on academic outcomes 
through three years, to assess whether these large effects persisted and whether they continue to 
match the magnitude of the effects found in New York. The report also examines the implemen- 
tation of the full program at the three colleges and presents a cost-effectiveness analysis, deter- 
mining the outcomes achieved per dollar spent. 


In sum, the programs were generally implemented as planned over the three-year period, 
with a few exceptions. Students received the program messages about full-time enrollment and 
on-time completion, and they participated in advising, tutoring, and career services at higher rates 
than their control group counterparts. The programs also continued to have large positive effects 
on enrollment, school progress, and degree receipt, and newly emerging effects on the percentage 
of students who transferred to four-year colleges. The effects presented here are strikingly similar 
to those found for the original CUNY ASAP model in New York, suggesting that this set of 
comprehensive support services can be implemented and help students beyond these two states. 


Finally, providing comprehensive support to students is not cheap. The direct cost of the 
Ohio Programs was about $5,500 per program group member over the three-year period. How- 
ever, because they increased degree receipt, the programs had a lower cost per degree received. 
Nonetheless, colleges may struggle to sustain these services over the long term without funding 
support from the state or other sources, as demonstrated by the fact that only one of the three Ohio 
colleges is continuing its program, despite clear evidence of effectiveness. 


The Ascendium Education Group provided anchor funding for the demonstration and 
evaluation, supplemented by grants from a consortium of other philanthropies, including Amold 
Ventures, the Bill & Melinda Gates Foundation, the ECMC Foundation, the Ford Foundation, the 
Greater Cincinnati Foundation, Haile U.S. Bank Foundation, KnowledgeWorks, the Kresge 
Foundation, and Lumina Foundation. 


The Ohio Programs Model (as Designed) 


The Ohio Programs model (based closely on CUNY ASAP) is designed to address multiple po- 
tential barriers to student success simultaneously and to address them over three full years. 


Table 1.1 presents the main components. First, students receive three forms of support 
that are available in theory to all students but are enhanced in this model. For example, each 
student is required to meet with a program adviser twice per month in the first semester, with 
requirements in the later semesters varying depending on the adviser’s determination of the 


Table 1.1 


Ohio Programs Model Components 


Requirements and Messages 


¢ Full-time enrollment: Required in fall and spring; 
summer attendance encouraged and financially cov- 
ered 

¢ Taking developmental courses early: Encouraged 
consistently and strongly 

¢ Graduating within three years: Encouraged con- 
sistently and strongly 


Student Services 


Advising: Students required to visit adviser twice per 
month in first semester and as directed based on need af- 
ter that; adviser caseloads of no more than 125 
Tutoring: Students required to attend tutoring if taking 
developmental courses, if identified as struggling by fac- 
ulty member/adviser, or if on academic probation. 

Career services: Students required to meet with campus 
career services staff or participate in an approved career 
services event once per semester 


Financial Support 


¢ Tuition waiver: Any difference between financial aid 
and tuition and fees waived 

¢ Monthly incentive: Monthly $50 gas/grocery gift 
card, contingent on participation 

¢ Textbook assistance: Voucher to cover textbook 
costs through the campus bookstore 


Course Enrollment 


Blocked courses and consolidated schedules: Seats 
held in specific sections of general or developmental edu- 
cation courses for program students during the first year; 
early registration for program students 

First-year seminar: New students required to take a stu- 
dent success course in the first semester, ideally in a sec- 
tion with other program students 


Management and Staffing 


¢ Program management: Local within each college, with periodic meetings and data sharing among members of the 
Ohio ASAP Network 

¢ Dedicated staffing: Fully dedicated program staff led by a director who reports to the provost or another senior leader 
at the college 


SOURCE: Program model information from the study colleges. 


student’s need. Students determined to be “high-need” are still required to meet twice per 
month, while other students could meet with their advisers less often. Program advisers also 
have lower caseloads than is typical. At least three hours per month of tutoring are also required 
for students in developmental education courses. 


Second, students receive three forms of financial support. They receive a tuition waiver 
that fills any gap between their existing grant financial aid and tuition and fees. They also receive 
financial assistance to cover the costs of textbooks, and a monthly gift card of $50, designed to 
help students purchase groceries or gas and to serve as an incentive to meet other program re- 
quirements (for example, attending advising). 


Third, the model is designed to help students with course enrollment by offering blocked 
courses to ensure program group students take courses together, a seminar course for first-year 
students, and early registration. 


In return for these services, students are required to attend full time and encouraged to 
take summer courses and complete developmental education requirements early, all with the aim 
of graduating within three years. 


Each college administered, managed, and staffed its own program. Each college’s team 
included a program director who supervised staff members and managed daily activities and 
program reporting, advisers who delivered core program services, and a program assistant. 
These staff members were employed by the program and worked primarily with students in the 
program. Each program director received guidance on program operations from a senior college 
administrator. 


Staff members also used a management information system designed specifically for the 
programs to manage the programs and track student participation and academic outcomes. !! 


The Evaluation 


To assess the effects of the programs, the evaluation targeted low-income students eligible for 
Pell Grants at the three Ohio colleges.'* Eligible and interested students were assigned at random 
using a lottery-like process to either a program group eligible to receive the programs’ services 
and benefits or a control group eligible to receive their college’s regular services. The differences 
between these two groups’ outcomes represent the estimated effects of the opportunity to partic- 
ipate in these programs. Random assignment ensures that student characteristics are not system- 
atically different at the start of the study, allowing differences in later outcomes to be attributed 
to the program." 


Evaluation Components 


The main goals of the evaluation are: (1) to assess the extent to which non-CUNY col- 
leges can implement a program based on CUNY ASAP; (2) to estimate the effects of the oppor- 
tunity to participate in the program; and (3) to calculate the programs’ costs and determine how 
they relate to the effects. To that end, the evaluation consists of three components: 


Implementation analysis. The evaluation explores how the Ohio Programs were imple- 
mented at the three colleges in the study and how the services that the programs offered compared 
with the usual college services. The study also examines participation in various services by pro- 
gram and control group members, as reported on a student survey, and the nature of the service 
contrast, or the difference between the services received by the program group and the services 
received by the control group. The implementation analysis aims to help researchers understand 
how the program produced the effects it did and, more broadly, to serve as a resource for other 


‘1A management information system is a database that holds information on program operations and that 
can produce reports on every level of a program’s management. 


More information on eligibility criteria is presented in Chapter 2. 


These differences are tested for statistical significance to assess the likelihood that they may have arisen 
by chance. 


colleges interested in how the Ohio Programs were implemented. '* Although the evaluation can- 
not determine which components of the model matter most, the implementation and survey find- 
ings may shed light on that issue. 


Impact analysis. The evaluation examines the Ohio Programs’ estimated effects on stu- 
dents’ academic progress and completion: their persistence in college, credit accumulation, de- 
gree receipt, and transfer to four-year universities. The study also analyzes effects for selected 
subgroups of students. 


Cost-effectiveness analysis. Finally, the evaluation examines the costs of the Ohio Pro- 
grams. Because the model provides many services, it is likely to be more expensive than most 
other community college programs. But its effects might also be larger. The cost-effectiveness 
analysis helps to compare the costs of different programs by considering program costs relative 
to program effects. 


Data Sources 


The evaluation of the Ohio Programs relies on several data sources. 


Baseline survey. Before random assignment, students filled out a baseline information 
form: a survey to collect baseline demographic information and other background information. 
This information is used to describe the sample, document that the characteristics of program and 
control group members were similar at the outset of the study, and to define subgroups of interest. 


College records. Measures of academic outcomes were obtained from detailed college 
transcript records and placement exam data, all of which were provided to MDRC by the three 
participating colleges. 


National Student Clearinghouse. Data from the National Student Clearinghouse, which 
covers student enrollment in nearly all postsecondary institutions in the United States, were used 
to examine academic outcomes such as enrollment, transfer, and graduation rates. 


Field research. Over the course of the evaluation, the evaluation team conducted field 
research visits to the three participating colleges to interview the administrators and staff members 
involved in the program. The interviews provided detailed information on the operations of the 
program. Students in the program and control groups were also interviewed about their perspec- 
tives on the college and the services available to them. 


One-year student survey: A survey was administered to all sample members approxi- 
mately one year after they were randomly assigned.!° The survey covered topics such as sample 
members’ participation in and experiences with student services, expectations of and engagement 
in college, employment, and financial aid and other financial issues. The survey data are used to 


Colleges interested in adopting the CUNY ASAP model can consult Boykin and Prince (2015). 


'SThe analysis plan for the study was registered on the Open Science Framework maintained by the Center 
for Open Science, and is available at https://osf.i0/2rSta. 


‘The response rate to the survey was 68 percent. See Appendix A for an analysis of survey response. 


help describe the differences between the program and control groups in the experiences they had 
and the services they received. 


The Organization of This Report 


Chapter 2 presents information on the colleges and students participating in the evaluation. Chap- 
ter 3 presents an in-depth analysis of the implementation of program components. Chapter 4 uses 
survey data to present participation in and receipt of services by the program and control group 
students. Chapter 5 assesses the Ohio Programs’ effects on students’ academic outcomes, includ- 
ing enrollment, credit accumulation, degree receipt, and transfers to four-year institutions. Chap- 
ter 6 presents a cost-effectiveness analysis. Chapter 7 discusses the Ohio Programs and CUNY 
ASAP, comparing the implementation of the two programs, the students who participated, the 
effects, and the costs. Finally, Chapter 8 concludes the report with a summary of findings and 
lessons. 


Chapter 2 


Participating Colleges and Students 


The Ohio Programs were evaluated at 3 of the 23 community colleges in the state. The colleges 
differ in terms of where they are located (in large cities or small cities), in their size and the num- 
ber of campuses they have, and in the type of students they serve. The characteristics of the stu- 
dents participating in the study reflect this diversity. This chapter presents characteristics of the 
participating colleges and students. 


The Colleges in the Demonstration 


The evaluation was conducted at Cincinnati State Technical and Community College, Cuyahoga 
Community College, and Lorain County Community College. Throughout the report, the colleges 
are referred to as C-State, Tri-C, and Lorain, respectively. 


These colleges were selected based on administrators’ support for the demonstration, the 
sizes of their student populations (to ensure an adequate sample of students for the study), and 
their willingness to participate in a randomized controlled trial. 


Table 2.1 presents a profile of the colleges. C-State and Tri-C are in the large cities of 
Cincinnati and Cleveland, respectively, while Lorain is in the smaller city of Elyria, in an outlying 
county of the Cleveland metropolitan region. The sample size and funding considerations led to 
the selection of three colleges with relatively large student populations. Tri-C, for example, has 
four campuses around Cleveland’s surrounding communities and has a total enrollment of around 
23,000 students. (The study took place at two of Tri-C’s four campuses.) C-State and Lorain serve 
8,000 and 11,000 students, respectively. Most other community colleges in the state have total 
enrollments of under 10,000 students.' Despite differences in size, the colleges are fairly typical 
for the state in terms of the characteristics of their student bodies. Most students attend part time, 
for example, a significant share are ages 25 or older, and about a third receive Pell Grants. 


Participating Students 


Recruitment for the evaluation took place before the spring 2015, fall 2015, and spring 2016 se- 
mesters.” Students were invited to participate through letters, emails, and phone calls. Those who 
were interested attended an intake session on campus, during which program staff members 


'U.S. News & World Report (2019). 

The program was pilot tested during the spring 2015 semester at all three colleges. At one college, MDRC 
randomly assigned students to the program group or control group for the spring 2015 pilot test. These students 
are included in the evaluation along with the students randomly assigned at all three colleges before the fall 2015 
and spring 2016 semesters. 


Table 2.1 


ASAP Ohio Demonstration College Profiles 


Characteristic Cincinnati State Lorain County Cuyahoga 


Technical and Community College Community 
Community College College 
‘Campus structure ~~ 1 maincampus+3 1 maincampus +5 4 main campuses + 
other locations satellite centers 8 satellite locations 
around the city across the county across the county 
Number of students 8,807 11,042 23,900 
Full-time students (%) 26 27 31 
Male (%) 44 39 40 
Black (%) 24 9 25 
Other minorities (%) 17 18 20 
Under 24 years of age (%) 55 70 61 
Campus setting City: large City: small City: large 
Geographic location Southwest Ohio Northeast Ohio Northeast Ohio 
Graduation rate (%) 15 23 15 
Undergraduate students receiving Pell Grants (%) 33 36 38 


Sample size (total = 1,501) 467 513 521 


SOURCES: Study colleges' websites and MDRC calculations using data from the Integrated Postsecondary 
Education Data System. 


described the program and evaluation. Students who agreed to take part in the study completed 
an informed consent form and a baseline information form. 


After completing the forms, each student was randomly assigned (using a computer pro- 
gram at MDRC) either to the program group, whose members had the opportunity to participate 
in the Ohio Programs, or to the control group, whose members had the opportunity to receive the 
usual college services. To thank them, students who were randomly assigned to either group re- 
ceived a gift card to compensate them for their time and transportation costs. 


Participation in the study was open to a broad group of students. Eligible students were 
required to be: 


e from low-income families (that is, they had to be eligible for Pell Grants); 


e seeking degrees; 
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e willing to attend full time; 
e majoring in degree programs that could be completed within three years;? and 
e newly enrolled or with 24 or fewer credits earned. 


Over the three semesters, the study enrolled a total of 1,501 students (806 in the program 
group and 695 in the control group). 


Table 2.2 presents selected characteristics of the students in the study. For comparison, 
characteristics of the student body at the three colleges and of community college students na- 
tionwide (where available) are also shown.* About 64 percent of study participants are women, 
and their average age when they joined the study was 23. Several characteristics are sometimes 
used to define students as “nontraditional,” or differing from the profile of a student entering 
college straight from high school.* The definition often includes older students (31 percent of 
the Ohio sample was 24 or older), those with children (27 percent of the Ohio students had 
children at study entry), and those working full time (a majority of the Ohio students were 
working, although most of them were working part time). For this study, nontraditional students 
are defined as those who were over age 24, working full time, had children, or had not received 
high school diplomas (having received General Educational Development [GED] certificates 
or other high school equivalencies instead). Using this definition, about half of the study sample 
is nontraditional. 


Three out of four students had developmental education requirements at study entry, 
meaning that they needed to complete at least one developmental course (in math, reading, or 
writing) as they progressed through college.° Finally, about a third of the participants entered the 
study as new students, with no credits earned, while another third had already earned 13 or more 
credits. 


The students participating in the evaluation look roughly similar to those nationwide, 
with some exceptions. Nationwide and in the evaluation sample, most students are women, about 
a third have children, most are working, and around three out of four have developmental educa- 
tion requirements. Students nationwide, however, are older on average than those in the Ohio 
sample and are more likely to work full time. There are some differences in race/ethnicity as well, 


3The requirement that a degree could be completed within three years meant that students majoring in certain 
fields were excluded from the study. The biggest of these fields was allied health (where students could receive 
degrees to become, for example, physician’s assistants, nurses, dental hygienists, etc.). This field represents a 
significant share of community college majors nationwide — about 23 percent. See Arbeit and Horn (2017). 

‘Appendix Table B.1 presents characteristics for the program and control group students, illustrating that 
random assignment was conducted properly and that there were few statistically significant differences between 
the two groups when the study began. 

>For example, see Radford, Cominole, and Skomsvold (2015). 

®Students who enter community college with developmental education requirements are less likely to grad- 
uate within six years than those without developmental requirements. See Chen and Simone (2016). 
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Table 2.2 


Characteristics of the Evaluation Sample, Study Colleges, and 
National Community College Students 


Evaluation Study National 

Characteristic Sample Colleges Sample 
Gender (%) 

Male 36 4 44 

Female 64 59 56 
Age (%) 

19 or younger 47 34 - 

20 to 23° 22 28 - 

24 or older® 31 38 44 
Average age (years) 23 - 28 
Race/ethnicity” (%) 

Hispanic 10 7 19 

White 46 60 56 

Black 35 21 16 

Other® 10 10 9 
Living with parents (%) 58 - - 
Has children (%) 27 - 32 
First person in the family to attend college (%) 34 - - 
Currently employed (%) 60 - 69 
Currently employed full time (%) 26 - 33 
Nontraditional® (%) 47 - - 
Has developmental education requirements (%) 74 - 68 
Number of credits earned, at study entry 

None 33 - - 

1 through 12 29 - - 

13 or more 38 - - 
Sample size 1,501 43,749 


SOURCE: MDRC calculations using baseline information form data and placement test data 
from the study colleges, and data from the Integrated Postsecondary Education Data System 
(IPEDS); Arbeit and Horn (2017); Chen and Simone (2016). 


NOTES: Distributions may not add to 100 percent because of rounding. 

Hyphens indicate unavailable data. 

aData for all students at the Ohio colleges require that the last two age categories be 20-24 
and 25 or older. 

bRespondents who said they were Hispanic and chose a race are included only in the 
"Hispanic" category. 

°The "other" category includes Asian/Pacific Islander, Native American/Alaska Native, 
multiracial, and other races and ethnicities. 

dNontraditional students are defined as those who were 24 or older, worked 35 or more hours 
per week, had children, or did not receive a high school diploma and were not enrolled in high 
school at the time of random assignment. Students are listed as nontraditional if they fit any of 
these characteristics. 
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with black students making up a higher percentage of the study sample than they do of the student 
population nationwide. The difference in age between the study sample and the national average 
may be partly due to the study’s eligibility requirements, since these differences are also apparent 
for the study sample compared with the overall student body at the three colleges. In particular, 
the requirement to attend full time may have discouraged older students from applying, if they 
were less likely to view full-time attendance as possible. 
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Chapter 3 


The Implementation of the Ohio Programs 


Chapter | presents the Ohio Programs model as it was designed or intended. This chapter de- 
scribes how the Ohio Programs were implemented over the study period (2015 to 2018).! The 
findings in this chapter are primarily based on MDRC field research and analyses of data from 
the colleges’ management information system that was used to track student participation. Chap- 
ter 4 discusses how students’ experiences in the Ohio Programs compare with the experiences of 
students in the control group, who received their colleges’ standard services. 


Summary of Findings and Organization of the Chapter 


The Ohio Programs were generally implemented as intended, but participation in some services 
was lower than expected. Additional findings include: 


e The programs were managed and staffed as planned, with some adjustments 
made to program staff members’ responsibilities to streamline program 
operations. 


e Thecolleges provided to program group students most of the planned program 
components, including messages about program requirements and recommen- 
dations, financial support, and student services. Blocked course schedules 
were not offered as planned. 


e Students participated in different services at different rates and also partici- 
pated at different rates from semester to semester. Enrolled students consist- 
ently participated in advising at high rates over the course of their three years 
in the programs. Lower percentages met the tutoring and career services re- 
quirements. However, the Ohio Programs made adjustments to improve par- 
ticipation, and a significant proportion of students participated in all three 
services. 


This chapter first describes the early implementation of the Ohio Programs, then moves 
to a discussion of the administration, staffing, and management of the programs. It then describes 
the implementation of and student participation in the various program components, covering 
requirements and messages, student support services, financial support, and course enrollment. 
Although each component is described separately, the Ohio Programs are an integrated package 
of services. The research study cannot disentangle the effects of each individual component. The 
chapter concludes with lessons gleaned from studying the implementation of the Ohio Programs. 


‘For the most part, this chapter uses the past tense to describe program implementation, although much of 
what is discussed may be true of the current program operating at Lorain County Community College. 
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Implementing Programs Based on the Original Model 


After agreeing to join the Accelerated Study in Associate Programs (ASAP) Ohio Demonstration 
in 2014, the three colleges each formed their own planning committees that worked closely with 
the City University of New York (CUNY) and MDRC to implement their programs. These plan- 
ning committees, which included administrators and staff members from the colleges’ academic 
affairs and student services departments, played an important role in launching the Ohio Pro- 
grams. The planning committees and other staff members involved in the early stages of the pro- 
grams identified match funding, identified or hired the rest of the staff, secured office space, and 
worked to put each program component into operation. 


The three colleges pilot tested their programs during the spring 2015 semester with small 
groups of students, so that they could refine the programs before full implementation and evalu- 
ation began.” Students who were randomly assigned to the programs were eligible to receive the 
services and support for up to three years after entering the study; most students began to receive 
full program services in the fall 2015 or spring 2016 semester. Outside of the evaluation, the Ohio 
Programs took in additional students during the fall 2016 and spring 2017 semesters to maintain 
consistent enrollment levels, as some students stopped attending the programs. These additional 
students helped to maintain the Ohio Programs’ staff-to-student ratio in case the colleges wanted 
to continue the programs after the study ended. Ultimately, one college decided to sustain and 
expand its program, and this college continued to recruit students, enrolling them for the fall 2017 
and fall 2018 semesters.? Figure 3.1 depicts the implementation timeline.* 


CUNY and MDRC helped the colleges adopt and develop their programs. CUNY led a 
workshop on the program model, helped the colleges develop action plans for implementation, 
and provided regular and continuing direct technical assistance and training in the ways each 
program component could be implemented. MDRC provided operational assistance and con- 
structive guidance during the pilot period to strengthen the programs. MDRC also created a man- 
agement information system based on the program model to collect and manage program partic- 
ipation data and, with CUNY, gave assistance to the colleges as they learned to use the data to 
monitor and improve program outcomes. 


The sections that follow describe how each component of the Ohio Programs model was 
implemented: program management, requirements and messages, student services, financial sup- 
port, and course enrollment. 


“At one college, students were randomly assigned to a program group who participated in the pilot program 
or to acontrol group who received the college’s standard services. These students are included in the evaluation 
sample along with the students randomly assigned at all three colleges before the fall 2015 and spring 2016 
semesters. At the other two colleges, students who participated in the pilot test were not selected by random 
assignment and are not a part of the evaluation sample. 

3See Chapter 8 for additional information on how the Ohio colleges are building on the ASAP Ohio 
Demonstration. 


4Each “cohort” in the figure is a group of students who entered the study at the same time. 
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Figure 3.1 


Implementation Timeline 


Evaluation period: Colleges implement the full program; students receive three 
years of services. These cohorts are included in MDRC’s evaluation: 


Colleges put the program e Cohort 1: spring 2015 - fall 2017 (one college only) 

into operation and pilot e Cohort 2: fall 2015 - summer 2018 (all colleges) 

test it e Cohort 3: spring 2016 - fall 2018 (all colleges) 

2014 2015 2016 2017 2018 
All colleges enroll One college expands 
small cohorts to program, enrolls over 
maintain staffing 200 new students 
levels 


SOURCE: Study colleges. 


Program Management 


The Ohio Programs model prescribed program management that included a dedicated program 
staff, the use of data for program improvement, and data sharing among the colleges in the ASAP 
Ohio Network. The study finds that the Ohio Programs were mostly managed and staffed as 
planned, with a few changes from the model related to the responsibilities of program staff mem- 
bers and the frequency with which the colleges in the ASAP Ohio Network met. 


Dedicated Staffing 


Each Ohio college administered, managed, and staffed its own program. At each college, 
a senior college administrator (such as the dean or vice president of student services, enrollment, 
student success, or academic affairs) oversaw the program. These senior administrators formally 
or informally supervised the program directors and offered them guidance on program implemen- 
tation and operation. They also advocated to executive leaders to secure resources the programs 
needed. The main program team consisted of: 


e Program director. The program director oversaw daily program activities, 
supervised a program’s staff, led recruitment, and managed reporting on stu- 
dents’ participation and academic outcomes.* Each program had one director, 


SAt one college this position was titled “program coordinator.” This report uses the term “director” 
throughout. 


17 


except for one college, which had two — one for each campus that operated a 
program. 


e Program advisers. Each program had advisers who delivered the main pro- 
gram services by meeting regularly with students.° Due to institutional policies 
at two of the colleges, the program advisers there focused on providing aca- 
demic coaching and personal support to students. Students received help reg- 
istering for courses primarily from the colleges’ general academic advisers. At 
the third college, the program advisers were considered part of the general ad- 
vising staff and provided registration services to students in addition to aca- 
demic and personal coaching. At all three colleges, the advisers helped recruit 
students and monitored student participation in the programs’ services. 


e Program assistant. Each campus had a program assistant to aid in administra- 
tion and coordination. The program assistant helped with entering data, report- 
ing, answering questions from students who walked into the program office, 
and providing advising appointment reminders to students. 


These staff members were official employees of the programs and worked primarily, if 
not solely, with program group students. The colleges did not employ staff members to provide 
tutoring and career services only to program group students. Instead, students received tutoring 
and career services mostly from existing centers on campus. 


e The responsibilities of some program staff members changed over time to 
streamline program operations. 


At some colleges, the program advisers took on additional responsibilities beyond the ad- 
vising that was originally part of the Ohio Programs model. At all three colleges the advisers pro- 
vided some career counseling, and at one college the advisers assisted with tutoring. These changes 
were meant to make the services more readily available and to increase student participation. 


Two of the colleges also ended up training advisers in financial aid systems and reporting. 
During the pilot test and early implementation periods, it was difficult for the financial aid office 
to keep abreast of tracking and allocating tuition waivers and textbook vouchers, on top of the 
office’s normal workload. Furthermore, if students had questions about their financial aid awards, 
advisers could not help immediately but had to escort the students to the financial aid office. To 
reduce the workload of the financial aid staff and streamline support for students, two colleges 
eventually trained program staff members in their programs’ financial aid processes. 


e Program staffing was relatively stable at two colleges; one campus at one 
college experienced significant turnover. 


®At one college these staff members were referred to as “coaches.” This chapter uses the term “advisers” 
throughout. 
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Stability in administration is often important to successful program implementation, so it 
is important to note that over the course of the three years the colleges operated their programs, 
all the colleges experienced some changes in senior college administrators who were not directly 
managing the programs, and two of the colleges experienced some minimal turnover of program 
staff members directly involved with the programs. One campus at one of the colleges, however, 
experienced significant turnover at all levels — the college eventually had to identify new senior 
administrators, a program director, and program advisers. Staff members at the college reported 
that this disruption reduced student participation; however, new staff members worked to rebuild 
students’ relationship with the program, and although the conclusions are not definite, the pattern 
of findings suggests that this staff turnover did not influence the program’s effects. (Academic 
outcomes are discussed in Chapter 5.) 


Data Management and Sharing 


In addition to a dedicated staff, the Ohio Programs model included data management and 
data sharing through the ASAP Ohio Network for the purposes of continual improvement. 


e The Ohio programs closely monitored student participation in services 
using the management information system developed for them. Staff 
members also tracked long-term outcomes using their colleges’ existing 
information systems. 


While developing the programs, the planning committees and program staff members 
established benchmarks for student participation in the model overall and specifically in advising, 
tutoring, and career services. To assess whether these benchmarks were being met and to manage 
the programs, staff members tracked and monitored data on students’ participation in the program 
and on their academic outcomes. Initially, the colleges used various systems to track participation 
data. One college used an Excel spreadsheet, another used an Access database, and the third used 
the management information system designed by MDRC. Over time, all of the programs transi- 
tioned to the management information system, which could handle large amounts of data better 
and which allowed staff members to enter data directly and produce reports. 


In interviews, the program directors reported that the management information system 
was useful for tracking student participation in the various program components (specifically, 
advising, tutoring, and career services) and completing monthly participation reports. Tracking 
and reporting on these data allowed the programs to identify areas for improvement. For example, 
advisers from one college described how partway through the month, they used the management 
information system to see who had not fulfilled their required advising appointments, so they 
could reach out to those students. 


While the management information system was useful for daily program management, 
staff members said that it could not be used to analyze and report on long-term outcomes such 
as persistence and graduation, since it was not directly connected to their colleges’ databases. 
To review long-term outcomes, programs relied on their colleges’ existing student information 
systems. 
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e The ASAP Ohio Network helped the Ohio Programs share lessons, but 
over time, the network met less regularly. 


While the three programs were managed separately, the Ohio Department of Higher 
Education (ODHE) worked with CUNY and MDRC to form the ASAP Ohio Network; the 
network was meant to help the colleges coordinate and share their progress, successes, and 
challenges with each other. Initially the network met quarterly, but over time it came to meet 
less regularly. During network meetings, staff members from ODHE, the program directors, 
and some senior administrators would meet to discuss the programs, review metrics such as 
program participation and academic outcomes, and share strategies for improvement. One di- 
rector noted in an interview that the ASAP Ohio Network provided a good opportunity to meet 
and collaborate with other program directors. 


Requirements and Messages 


The Ohio Programs model included specific requirements and messages. Students in the Ohio 
Programs were required to enroll full time in the fall and spring semesters and were encouraged 
to enroll in summer courses. Students were also strongly encouraged to take any developmental 
education courses they needed early in their academic careers, and were informed that they should 
plan to graduate within three years. 


According to interviews with program staff members and students, program advisers em- 
phasized these requirements and messages. The messages were discussed during students’ initial 
orientation to the programs and reinforced during advising sessions. Compliance with the full- 
time requirement was monitored through registration holds that required students to see their ad- 
visers before making changes to their schedules, and was encouraged to a certain extent though 
the financial support, as students who enrolled part time were not eligible for all of the financial 
support (though they could still receive other services). Program advisers reported that the full- 
time enrollment requirement was difficult for some students due to work and family responsibil- 
ities. On the other hand, the programs did encourage and enable more students to enroll full time. 
(Chapters 4 and 5 present additional detail on students’ receipt of and compliance with the Ohio 
Programs’ requirements and messages. ) 


Student Support Services 


The Ohio Programs provided students with advising, tutoring, and career services as planned. 
Although students were informed that participation in the services was mandatory, program group 
students were more likely to meet advising requirements than they were to meet tutoring and 
career services requirements. Over time, to increase participation, programs modified the require- 
ments related to tutoring and career services, and also modified the ways students could fulfill 
those requirements. 
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The sections below describe the implementation of and student participation in each ser- 
vice component. (Appendix Table C.1 shows participation in advising and career services by 
semester, and Appendix Table C.2 shows participation in tutoring by semester.) 


Advising 


The Ohio programs implemented advising as planned. Each college employed advisers 
who worked with relatively small caseloads of students. Advisers’ caseloads varied by semester 
and by college, ranging from 80 to 140 students per adviser. Students were informed that advising 
was mandatory and were required to meet with their advisers twice each month during their first 
semester in the programs. Starting in the second semester and extending through the end of the 
three-year program, advisers “triaged” students, or sorted them into low-, medium-, and high- 
need groups. Students in the high-need group continued to meet with their assigned advisers twice 
per month. Students in medium- and low-need groups had various requirements but were often 
required to meet with their advisers at least once per month. 


Advisers were typically full-time employees; one college added a part-time adviser to its 
staff for a brief time; however, the program eventually decided not to continue the position be- 
cause the adviser’s part-time availability made it difficult for the adviser’s assigned students to 
receive services. 


Management information system data show that student participation in advising was 
high. The percentage of enrolled students who received advising at least once in a given semester 
ranged from 79 percent in the lowest semester to 95 percent in the highest semester. Due to the 
“triage advising” model, the required number of advising appointments varied by semester and 
by student. In their first semester, about 61 percent of students met with their advisers at least six 
times, which means the students probably met the two required appointments each month that 
were required of them that term. To ensure consistent participation in advising, program staff 
members used a variety of strategies — such as calls, text messages, and posts on course- 
management systems — to connect with students and remind them of upcoming appointments.’ 
The advising sessions covered a range of topics including how classes were going, degree plan- 
ning, registration, and life outside of school. Program advisers also connected students with the 
counseling center, food banks, financial aid, and other services on campus as needed. 


In interviews, students emphasized that advising was an important aspect of the pro- 
grams. For example, one student explained: 


My adviser helps me schedule everything. She answers every question I have. She 
helps me pick out my books.... She actually took me to the food bank. She met me 
in the bookstore. She helps me with everything, just everything. If I have any ques- 
tions, she’s quick about messaging me back.... She helped. 


7Course-management systems are online portals where students can get assignments and material and sub- 
mit assignments. 
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Tutoring 


Program group students in developmental education courses were required to attend at 
least three hours of tutoring per month. In general, tutoring was provided by the existing centers 
on campus; however, at times, each program also offered special tutoring hours (usually in math) 
just for program group students. 


While tutoring was offered as planned, participation in tutoring was initially lower than 
expected and the percentage of students enrolled in developmental education who attended at 
least one tutoring session ranged from 38 percent to 72 percent, depending on the semester. It is 
important to note that the number of students enrolled in developmental education declined each 
semester as students completed their requirements or stopped enrolling in college. According to 
program staff members, it was difficult for students to meet the required number of tutoring hours 
due to other work and personal commitments. Staff members also reported that it was difficult to 
get students to understand the value of tutoring. 


To increase the proportion of students meeting their tutoring-participation requirements, 
the programs waived the requirements for students who were receiving As or Bs on course as- 
signments and exams. They also increased the number of avenues through which students could 
receive tutoring. For example, instead of going to tutoring centers on campus, students could use 
online tutoring resources or receive assistance from a family member. Because in later semesters 
only a small proportion of students still had to complete developmental education courses and 
because tutoring requirements were waived for high-performing students, it is difficult to gauge 
fully the extent to which students complied with the tutoring requirements in later semesters. Staff 
members reported that increasing flexibility in tutoring options increased participation. 


Career Services 


Program group students were required to complete one career-services activity per se- 
mester. Students could fulfill this requirement in numerous ways, such as meeting with a career 
services staff member, attending a career-related workshop, or completing a career assessment. 
Each of the colleges typically had a required activity for first-semester students. After the first 
semester, students could select their activities. At two colleges, students could work directly with 
their program advisers instead of a career or employment specialist to complete the career- 
services requirement. At all colleges, advisers played a large role in tracking participation by col- 
lecting information on participation from students or the career center staff. 


Early on, program staff members noted that it was difficult to get students to complete 
the career-services requirement. In the first semester, only 45 percent of enrolled students met 
with a career specialist. The Ohio Programs used various strategies to increase participation. For 
example, one program assigned each student a specific month in which to complete the career 
requirement, rather than asking them all to complete the requirement before the end of the semes- 
ter. Participation in career advising increased after the first semester, with participation ranging 
from 58 percent to 69 percent of enrolled students, depending on the semester. 
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Course Enrollment 


The Ohio Programs model included two types of changes to students’ course-enrollment patterns: 
(1) blocked courses and consolidated schedules and (2) a first-year seminar. 


e The Ohio Programs did not implement blocked course schedules as 
planned but attempted to place program group students in courses 
with each other. The first-year seminar was offered as part of general 
college services, and one-third of students participated after entering 
the programs. 


The Ohio Programs were unable to implement blocked course schedules as planned. 
College administrators and staff members reported that it was difficult to identify appropriate 
courses for block scheduling since students could have up to 24 credits when they entered the 
programs. Many students had already completed various developmental or general education 
courses that would have been included in a blocked schedule, making it difficult to reserve a 
class section just for program group students.* Staff members also reported that it was difficult 
to identify appropriate times for the blocked courses since students took classes at various times 
throughout the day. 


Although the colleges did not officially implement blocked courses, program advisers 
attempted to promote a sense of belonging among students (one of the theorized benefits of 
blocked courses) by guiding program group students into courses with other program group stu- 
dents. Some advisers also informed students when there were others students from the program 
in their courses. About half of program group students were enrolled in at least one course with 
at least four other students in the program group. The next chapter explores whether students in 
the program group experienced a greater sense of belonging than control group students, using 
data from a student survey. 


Regarding the first-year seminar, the colleges already offered similar courses that intro- 
duced students to academic strategies such as note-taking and study skills. The colleges typically 
required this course for most if not all students, so the Ohio Programs did not report taking any 
specific measures to create first-year seminars or encourage students to enroll. Enrollment data 
show that 34 percent of program group students took a first-year seminar after joining the pro- 
grams. This number may reflect the fact that about one-third of study participants were incoming 
first-semester students when they began the programs. Students who joined the programs after 
their first semester in college may have already taken the seminar. 


8Colleges often require students to take non-major-specific courses such as math or English. These courses 
are referred to as “general education” courses. 
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Financial Support 


The Ohio programs offered students in the program group several forms of financial support (tu- 
ition waivers, textbook assistance, and monthly financial incentives) intended to help them en- 
gage in college and especially to enroll full time. 


e The Ohio Programs offered tuition and textbook assistance as planned. 
Notably, many students did not receive these forms of financial support 
from the programs because they had enough Pell Grant funding to cover 
their tuition and book expenses. 


The sections below describe how each type of financial support was implemented and 
how much funding students received. Chapter 6 provides more detailed information on the total 
cost of the financial support. 


Tuition Waiver 


The colleges provided program group students with waivers to cover the difference be- 
tween their tuition and fees and their grant aid for each fall and spring semester. Students also 
received tuition assistance for the summer terms, but not all colleges fully covered the balances. 


Table 3.1 shows the percentage of program group students who received tuition waivers 
each semester and the average waiver amount each semester. As shown, 57 percent of program 
group students received at least one tuition waiver over the course of the three-year program and 
each semester fewer than 30 percent of students in the program group received waivers. Many 
program group members probably received enough Pell Grant aid to cover their tuition and fees 
and thus did not need the programs’ tuition assistance. The average waiver amount was $107 per 
program group student (including those who did not enroll or did not need waivers). Among those 
who received waivers, the average waiver varied depending on the semester, ranging from $369 
to $675.° The average tuition waiver may be lower in later semesters due to several factors, in- 
cluding fewer students being enrolled in the programs. 


While many students did not need the tuition waiver, they still found it to be valuable. 
One student explained, “If I wasn’t able to get the gap tuition ... then I probably wouldn’t be able 
to stay [enrolled].” Even students who did not receive tuition waivers may have benefited from 
the security of knowing that their tuition and fees would be covered, as research shows that cov- 
ering students’ tuition can increase the likelihood that they enroll in college and enroll full time.” 


Textbook Assistance 


The Ohio programs provided students with financial assistance to cover the costs of text- 
books at their campus bookstores. At two colleges, students received $300 during each of the fall 


°Data from one college were unavailable for spring 2015, summer 2018, and fall 2018. 
‘0Headlam, Anzelone, and Weiss (2018); Dynarski, Libassi, Michelmore, and Owen (2018). 
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Table 3.1 


Tuition Waiver Receipt Among Ohio Program Group Students Over Three Years 


Outcome Mean 
Ever received a tuition waiver (%) 56.9 
Received a tuition waiver (%) 
First semester 23.7 
Second semester 27.0 
Third semester 15.8 
Fourth semester 13.1 
Fifth semester 6.2 
Sixth semester 2.8 
Average waiver received among those who received waivers ($) 
First semester 674.5 
Second semester 645.3 
Third semester 608.0 
Fourth semester 510.4 
Fifth semester 499.3 
Sixth semester 369.3 
Average waiver received among all students ($) 106.6 
First semester 160.1 
Second semester 174.1 
Third semester 96.3 
Fourth semester 66.6 
Fifth semester 30.8 
Sixth semester 10.4 
Total program group students 806 


SOURCE: MDRC calculations using financial aid data from the study colleges. 


NOTES: Tuition waivers received in spring and summer are combined. 


Data from one college were unavailable for the spring 2015, summer 2018, and fall 2018 terms. These students 
are not included in calculations of receipt rates for these semesters or for the cumulative measures "ever received 


a tuition waiver" or "average waiver received among all students." 


and spring semesters and $150 during the summer terms. At the third college students received 


all their books free of charge; there was no capped amount. 


As shown in Table 3.2, approximately 76 percent of enrolled program group students 
received textbook assistance at least once during their time in the programs. However, fewer than 
70 percent of enrolled students received textbook assistance each semester. On average, enrolled 
students who received assistance received $363 in the fall and spring terms and $268 in the sum- 
mer terms.'' Some enrolled students may not have received textbook vouchers because at two of 
the colleges, students only got the textbook vouchers after they exhausted their Pell Grant funds. 
The third college allowed students to use the textbook vouchers before applying their Pell Grant 


"Data from one college were unavailable for spring 2015, summer 2018, and fall 2018. 
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Table 3.2 


Receipt of Textbook Assistance Among Enrolled Students 


Program 
Outcome Group 
Ever received textbook assistance, among all program students (%) 75.7 
Received textbook assistance® (%) 
First semester 66.6 
Second semester 67.2 
Third semester 61.4 
Fourth semester 61.4 
Fifth semester 52.7 
Sixth semester 37.8 
Average textbook assistance amount received? ($) 
Fall/spring terms 363.3 
Summer terms 267.9 
Sample size 806 


SOURCE: MDRC calculations using financial aid data from the study colleges. 


NOTES: Data from one college were unavailable for spring 2015, summer 2018, and fall 2018. These 
students are not included in calculations of receipt rates for these semesters or for the cumulative 
measure “ever received textbook assistance.” 

Textbook assistance receipt for spring and summer are combined. 

aReceipt of textbook assistance is shown among enrolled students in each semester. 

bAverage textbook assistance amount received does not include students who did not receive awards. 


funds to books. While program group students at this third college could use the Pell Grant fund- 
ing that they saved to offset other expenses such as housing, the college spent more on textbook 
assistance. This fact suggests that providing students with textbook assistance after Pell Grants 
are applied is one way to control program costs, although doing so reduces the financial benefit 
to students. 


Monthly Incentive 


Program group students were eligible to receive a $50 incentive each month contingent 
on participation in advising, tutoring, and career services.'? Each campus dispersed gift cards to 
stores in its area where students could purchase groceries or gas.!> A large proportion of enrolled 


‘Program staff members and managers monitored participation using records of student participation in 
program activities as described above. 


Giant Eagle, Walmart, Kroger, etc. 
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incentive. About 43 percent to 53 percent of enrolled program group students received three or 
more incentives each semester. These percentages are expected, since the gift card was tied to 
meeting participation requirements each month. 


Program staff members and students reported that students spent the gift cards in various 
ways. Some relied on the gift cards for everyday expenses like food and gas, while others bought 
personal items, used the cards to offset the cost of large purchases like laptops, or did not use 
them. In the survey administered to students 12 months after they entered the programs, 74 per- 
cent of students reported spending the gift card on food and 54 percent reported spending it on 
gas (see Appendix Table C.3). One program adviser reported that “the money” gets students in- 
terested, but the advising keeps them involved. Another staff member thought the incentive was 
not large enough to motivate students to complete the program requirements. For many students, 
the monthly gift card was not the biggest factor leading them to attend program services; however, 
it did help them cover a range of personal expenses, including gas, tuition, and personal items. 


Students’ Experiences in the Programs 


e Almost all students in the Ohio Programs had positive experiences. 


On the student survey, about 92 percent of program group students rated their overall 
experience in the programs as “excellent” or “good” (see Appendix Table C.3). Only 8 percent 
of students rated their experience as “fair” or “poor.” This finding corroborates what program 
group students described in interviews. In those interviews, many program group students praised 
the programs as being essential to their college experience. As one student said: 


[The program] definitely has made my college experience a lot easier so far be- 
cause starting out my parents never went to college, so I have no clue [about] the 
lingo that goes [with] college and the different things. I had no clue how it was 
structured.... I needed somebody to explain to me how everything works and the 
requirements for different things. And so it’s definitely helped with that under- 
standing of how college works and what I need to do here and why I’m here, and 
it’s made things a lot easier. 


ASAP Ohio Demonstration Implementation Lessons 


The implementation of the program model in Ohio yielded the following lessons for other col- 
leges to consider when implementing this model or a similar one. 


e When a college implements a comprehensive student program, it may 
need to adjust program staff members’ roles and responsibilities to align 
with institutional policies. 


The roles and responsibilities of program advisers varied slightly among colleges due to 
differences in institutional context and policies. As described earlier, at two colleges students 
were required to see a general academic adviser for class registration. At the third college, how- 
ever, program advisers could advise students on registration in addition to providing program 
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services. This additional function limited the number of people program group students needed 
to coordinate with at that college. There is no evidence that this difference in advising structure 
resulted in differences in the program’s effectiveness; however, it may have contributed to long- 
term sustainability. While staffing responsibilities may need to be adjusted to align with institu- 
tional structures, programs can strive to maintain fidelity to the model by having one person serve 
as the main adviser or coach. 


e Programs need robust data-management systems that allow them to 
monitor long-term outcomes such as persistence and also shorter-term 
outcomes such as participation in advising, tutoring, and career services. 


A vital aspect of the Ohio Programs was collecting and reviewing participation data for 
each student in the program group. Doing so allowed the programs to identify students who were 
not meeting requirements and might have needed additional support. A management information 
system was created for the programs because program staff members had to monitor short-term 
outcomes such as participation in advising and tutoring. Programs found the management infor- 
mation system created for them to be useful for monitoring service use, but they still needed to 
use their colleges’ existing student information systems because the program management infor- 
mation system was not connected to those existing systems. It would be helpful if staff members 
could use the same system to monitor long-term and short-term outcomes. Staff members should 
also be able to enter and retrieve data easily so they can reach out to students regularly and im- 
prove the program continually. 


e Programs can allow students to fulfill requirements in various ways. 


Initially, the Ohio Programs had difficulty getting some students to meet the tutoring and 
career-services requirements. Participation increased as the programs gave students more options 
for fulfilling the requirements. These types of refinements could help programs accommodate 
different students’ needs while still maintaining their structures. However, it may be important to 
plan and monitor carefully how different requirements are filled, to ensure that students still re- 
ceive adequate services. 


e Blocked course schedules may work better for first-time students. 


The Ohio Programs found it difficult to implement blocked courses because many stu- 
dents had already completed certain developmental and general education courses that could have 
been reserved for program group students. This finding suggests that blocked courses may be 
more appropriate for new, incoming students. Additionally, it was difficult to reserve full course 
sections because students attended classes at different times of the day (morning, evening, etc.). 
Larger programs with more students may find it easier to create and fill course blocks at multiple 
times of the day. 
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Conclusion 


For the most part, the Ohio colleges were able to implement programs based on CUNY ASAP. 
During implementation, the colleges tweaked certain components to reflect their institutional pol- 
icles and to increase student participation in services. Students reported positive experiences in 
the programs, especially with the advising and financial support components, and the Ohio col- 
leges’ implementation experience offers lessons for other colleges interested in adopting the 
model. The next chapter presents how the experiences of the program group compared with those 
of their control group counterparts who received their colleges’ standard services. 
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Chapter 4 


The Ohio Programs’ Services Compared 
with the Colleges’ Usual Services 


The previous chapter described the Ohio Programs’ services. This chapter highlights the differ- 
ences between the experiences of students in the program group and those in the control group 
— the service contrast. This service contrast is essential to explaining any observed effects on 
academic outcomes described in Chapter 5. The findings in this chapter are based on three data 
sources: MDRC field research, the student survey, and college administrative records. 


Summary of Findings 


Overall, there is a noticeable service contrast between program group and control group student 
experiences in relation to all the program components described in Chapter 3. Findings include: 


e Two-thirds to three-quarters of program group students reported hearing the 
three primary program messages to enroll full time, enroll in summer courses, 
and graduate in three years, whereas fewer than half of control group students 
reported hearing the same messages. 


e Program group students met with advisers more than three times as much as 
control group students in their first two semesters. They also made greater use 
of tutoring and career services. 


e Program group students relied less than control group students on student 
loans, personal savings, or support from family and friends to pay for college; 
they also expressed less concern about their ability to pay for college. 


This chapter is divided into five sections; the first four represent components of the pro- 
gram model described in Chapter 3 and the last discusses students’ experiences. Tables for all 
analyses can be found in Appendix D. 


Requirements and Messages 


e Students in the program group were more likely than those in the control 
group to report that they were encouraged to enroll full time, to enroll in 
summer courses, and to graduate in three years. 


As Figure 4.1 shows, 74 percent of program group students reported hearing from faculty 
or staff members that it was important to enroll full time, compared with 46 percent of control 
group students. 
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Percentage of students who said they 


received the message 


Figure 4.1 


The Ohio Programs Pushed Completion-Related Messages to Program Students 
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You should enroll in school full You should enroll in school You should obtain an 
time during the summer semesters associate's degree within 3 
years 
Message 


mProgram group OControl group 


SOURCE: MDRC calculations using data from the student survey. 


NOTE: Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 


Students in the program and control groups also reported a difference in the messages 


they heard about summer enrollment. Sixty-six percent of program group students heard from 
faculty or staff members that it was important to enroll in summer courses, compared with 40 
percent of control group students. 


Program group students were also more likely to be encouraged to graduate within three 


years. Seventy-three percent of the program group reported hearing the message about graduating 
in three years, compared with 45 percent of the control group. 


Student Services 


Each Ohio college offered advising, tutoring, and career services to all students. The Ohio Pro- 
grams made use of these existing services. Interviews with college staff members provided insight 
into the services available to all students. 


Advising. As described in Chapters 1 and 3, advising was a central component of the 


Ohio Programs and program advisers, who worked with caseloads of about 125 students each, 
provided academic and personal support to their students. In contrast, students in the control group 
primarily received advising from existing advising centers on campus. Each college’s program 
had a different standard advising structure. At two colleges, advisers only addressed academic 
issues; other staff members on campus addressed career or personal matters. At the third college, 
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the same advisers addressed academic, career, and personal issues. It was not typical at any col- 
lege for control group students to have assigned academic advisers or mandatory advising. It was 
also typical for the student/academic adviser ratio to exceed 300 to 1. 


Tutoring. Each college offered a variety of tutoring options and a dedicated space where 
students could meet with tutors. One-on-one or group tutoring options were available in a range 
of subjects, although special support was available for math and writing. Each college also offered 
alternative options such as online or phone-based tutoring. These tutoring options were available 
to all students and students generally were not required to attend, with a few exceptions (some 
athletes and the students of certain instructors, for example). 


Career services. Multiple career services were available at each college. They included 
various one-on-one meetings and group workshops designed to help students to gain skills for 
their job searches. These workshops covered résumé development, tips for interviewing, mock 
interviews, success in a new job, networking, skills and interest inventories, and career explora- 
tion. Internships or cooperative education opportunities were also available through each college, 
and each also held career fairs.! Staff members from career services also made presentations to 
students in some classes. 


Multiple offices at each college provided career services. For example, at one college, 
a student may have gone to the career center to take a career-exploration assessment, and then 
taken the results to a counselor in the counseling center to talk about how to pursue that career. 
Career development services were also decentralized at a second college, where students may 
have received support from an employment center, from counseling services, or through coop- 
erative education. For the most part, students were not required to participate in any career- 
development activities, though all activities were available to all students. One exception was 
that at one college, many majors required students to complete at least one cooperative educa- 
tion placement to graduate. 


e Program group students were more likely to engage in advising, tutoring, 
and career services than control group students. The difference is partic- 
ularly notable for tutoring and career services. 


As shown in Figure 4.2, even though control group students reported in the survey that 
they met with advisers, went to tutoring, and met with career services staff members, program 
group students were much more likely to report in the survey that they engaged in these student 
services. The differences between the program and control group students are particularly distinct 
when it comes to tutoring and career services. 


Eighty-eight percent of program group students who responded to the student survey said 
that they had spoken with an adviser at least once in their first year of college, compared with 79 


‘Cooperative education, or “co-op,” provides academic credit for a structured job experience. At C-State, 
where co-ops were common, each co-op lasted one semester. 
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Figure 4.2 


The Ohio Programs Boosted Students’ Use of Support Services 
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NOTE: Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 


percent of control group students (though the question did not specify what type of adviser — 
academic, career, or personal). Furthermore, 63 percent of program group survey respondents 
reported ever receiving tutoring, compared with 45 percent of control group students; 62 percent 
of program group survey respondents met with a career or employment services staff member 
during the first year, compared with 28 percent of control group survey respondents. 


e Program group students reported more frequent participation in advis- 
ing, tutoring, and career services than control group students. 


Program group students were not only more likely to receive advising, tutoring, and ca- 
reer services, but also made much more frequent use of these services, as Figure 4.3 shows. In 
particular, students in the program group spoke with advisers many more times in both of the first 
two semesters than did students in the control group. On average, control group students reported 
meeting with advisers 4 to 5 times in each of the first two semesters, while program group students 
reported meeting with advisers 13 to 14 times. Along these lines, program group students reported 
going to tutoring more than twice as many times as the control group students. Furthermore, pro- 
gram group students reported seeing career services staff members at least four times as often as 
control group students: Control group students reported an average of 0.6 visits in each of the first 
two semesters, while program group students reported 2 and 3 visits in the first two semesters, 
respectively. 
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Figure 4.3 


Ohio Program Students Met More with Support Services 


16 


8.4*** 10.1*** 


Number of meetings 


Advising Tutoring Career Advising Tutoring Career 
services services 


First semester Second semester 


mProgram group OControl group 


SOURCE: MDRC calculations using data from the student survey. 


NOTE: Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 


e Program group students reported that they communicated with advisers 
in more varied ways than control group students did, spent more time in 
meetings with advisers, and were more likely to discuss personal matters. 


There is evidence that the program group students had different types of interactions with 
support staff members than control group students did. Among those who spoke with advisers, 
program group survey respondents communicated with their advisers in more varied ways than 
did students in the control group. Program group students reported mostly in-person meetings, 
phone calls, and emails, with some text messages and social media contact; control group students 
were less likely to have contact with advisers over the phone, by email, by text message, or over 
social media (see Appendix Table D.2). Program group survey respondents also reported spend- 
ing more time with advisers during each visit than did control group students. 


Survey respondents in both groups reported talking with advisers about a range of topics, 
including academic goals, course selection, and internships, but program group students reported 
covering more topics with their advisers. For example, as shown in Figure 4.4, nearly half of 
program group survey respondents who saw advisers reported discussing career planning, com- 
pared with just one-fourth of control group survey respondents who saw advisers. Notably, these 
program group survey respondents were much more likely to report discussing personal matters 
than the corresponding control group student survey respondents: 48 percent compared with 20 
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Figure 4.4 


Ohio Program Students Addressed Personal Matters and 
Other Topics More with Advisers 
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NOTE: Differences were not tested for statistical significance because these outcomes were reported 
only among students who spoke with advisers. 


percent. This difference may have come about because students who visit their advisers more 
times over the course of a year can form closer relationships with those advisers. 


Because these outcomes are reported only among students who spoke with advisers, they 
were not tested for statistical significance. However, these gaps are large enough to imply that the 
program group students’ advising experiences had more depth and breadth than those of the con- 
trol group. Several program staff members and students said that they felt advising was the most 
important nonfinancial component of the programs. In interviews, students described the unwa- 
vering support advisers provided in dealing with a range of personal and academic issues. As one 
student said, “You can definitely talk to them about anything. They make it real comfortable.” 


Course Enrollment 


The Ohio Programs did not make changes to pedagogy, curriculum, or anything else that hap- 
pened inside the classroom. Three things related to course enrollment were part of the model: a 
first-year seminar, priority registration, and blocked courses. 
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The program group students in this study experienced the same faculty and curriculum 
as the other students at the colleges, including the first-year seminar, which is a student success 
course taken in the first semester (described in Chapter 3). In fact, each college “mandates” that 
new students enroll. However, there are no mechanisms in place to enforce this mandate. An 
analysis of student enrollment records suggests that the same percentage of the program and con- 
trol groups took the course. There are no records available to confirm the number of students who 
registered early. 


e Although program group students were more likely than control group 
students to be enrolled in a course with other program group students, 
relatively few of them did share a course. 


Blocked courses were not a widespread practice across the colleges. Where they did exist, 
they were associated with degree programs for specific professions (such as occupational thera- 
pist or medical laboratory technician). According to the program model, blocked courses could 
facilitate a sense of community among program group students who took classes together. An 
analysis of the colleges’ enrollment data shows that a higher percentage of program group stu- 
dents enrolled in a course with at least four other program group students than control group 
students enrolled in a course with at least four other control group students. Yet fewer than one- 
third of program group students shared a class with at least four other program group students in 
the first semester and fewer than one in five did so in the second semester (see Appendix Table 
D.5). Since relatively few program group students enrolled in courses with a high concentration 
of other program group students, it is unlikely that this substitute for the blocked courses element 
of the Ohio Programs model had a substantial effect on program group students’ success. 


Financial Support 


The program provides three forms of financial support to students that are intended to ease their 
participation in college: tuition waivers, textbook assistance, and monthly incentives. While these 
forms of support did not eliminate all financial needs, evidence suggests that they made it easier 
for program group students to pay for college. They did not, however, reduce program group 
students’ need to work. 


Control group students had no tuition waiver equivalent to that available for program 
group students. Aside from federal financial aid, there is not much need-based support for stu- 
dents in the general population at the Ohio colleges; scholarships are primarily available to high 
performers. One need-based state program, the Ohio College Opportunity Grant (OCOG), which 
the Ohio Department of Higher Education administers, provides grant money to Ohio residents 
who demonstrate the highest levels of financial need. Additionally, community college students 
who have exhausted their federal Pell Grants may be eligible for OCOG to take a third term of 
courses (usually summer courses). 


As described in Chapter 3, program group students received financial aid to cover the 
cost of textbooks at the campus bookstore, and one college administered these funds differently 
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Percentage of students 


than the other two. While each college had several programs that offered funds to offset the cost 
of textbooks, these programs served a limited number of students and it is not likely that many 
control group students had access to these programs. Program group students were nearly twice 
as likely to report that they received all their textbooks free of charge than their control group 
counterparts (39 percent versus 19 percent — see Appendix Table D.6). 


As described earlier, program group students were eligible to receive a $50 incentive each 
month contingent on participation in advising, tutoring, and career services. Monthly incentives 
for student participation in activities were not otherwise common among the colleges. There are 
no data to show students in the control group had access to anything comparable. 


e Program group students reported that they relied less than control group 
students on personal savings or support from family or friends to pay for 
college. 


As Figure 4.5 shows, students in the program group were less likely than students in the 
control group to report that they relied on parents, relatives, partners, or friends to help pay for 
college. Program group students were also less likely than control group students to say that they 


Figure 4.5 


Ohio Program Students Relied Less on Savings 
and Family/Friends to Pay for College 
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relied on personal savings. These findings suggest that the programs’ tuition waiver and possibly 
other forms of financial support affected how some students financed their education, reducing 
their need to spend down their savings and receive help from others. 


e Fewer program group students than control group students expressed 
concerns about their ability to pay for college. 


As shown in Figure 4.6, 47 percent of program group survey respondents indicated they 
had few or no concerns about their ability to pay for college, compared with 33 percent of control 
group members. In fact, nearly one-third of control group respondents (32 percent) reported that 
they chose not to register for college full time because they could not afford it. Only 16 percent 
of program group students reported the same. Facilitating and supporting full-time enrollment is 
central to the program model, and these findings suggest that the programs made a big difference 
for students. 


One in five control group members (20 percent) reported that they often or very often did 
not buy textbooks because of the cost. The programs, through their textbook assistance, reduced 
that proportion to about one in seven program group students (14 percent). There was a small 


Figure 4.6 
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difference in the number of students reporting having missed class often or very often because 
they could not pay for transportation: 11 percent of control group students reported having to miss 
class for this reason, compared with only 8 percent of program group students. 


e Two-thirds of students in both research groups worked during their first 
two semesters, and they worked about the same number of hours per 
week. 


There were no differences between the program and control group in the percentage of 
students reporting they had a job in the first year; about two-thirds of both groups reported 
having been employed, as shown in Figure 4.6. There is also little difference in the average 
hours worked in the first year among students who did work: around 30 hours a week (see 
Appendix Table D.7). This finding is important because it supports the assertion that a large 
portion of the study sample are nontraditional students, and being a nontraditional student is 
often a risk factor for a lack of postsecondary success.” Furthermore, the finding suggests that 
any later effects on educational outcomes did not occur because the programs enabled students 
to reduce the time they spent working. 


Students’ Experiences in College 


The student survey did not only ask about students’ use of and experiences with various services 
and forms of support. Both program and control group students were also surveyed about their 
overall experiences in college. 


e Program group students generally reported more favorable college expe- 
riences relating to social integration and support services than control 
group students. 


Figure 4.7 presents effects on four measures of engagement in school. On a scale that 
measures integration and sense of belonging, for example, program group students were a little 
less likely to report having a low sense of integration and belonging than control group students 
(14 percent compared with 20 percent). However, the programs did not increase the proportion 
of students who reported having a high sense of integration and belonging. 


Program group students were more likely to report that they had the support they needed. 
Eighty-nine percent of program group students reported having a college employee to turn to for 
advice, compared with 72 percent of control group students. Further, 91 percent of program group 
students reported having all or most of the services that they needed to succeed, compared with 
81 percent of control group students. 


*Horn and Carroll (1996). See also Sommo and Ratledge (2016). 
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Figure 4.7 
Ohio Program Students Were More Engaged 
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Conclusion 


A number of service contrast differences are highlighted in this chapter. In areas related to every 
program component, this analysis indicates that program group students’ experiences were dra- 
matically different from those of control group students. The next chapter presents the effects 
these service differences had on students’ academic outcomes. 
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Chapter 5 


Effects on Academic Outcomes 


The earlier brief from this demonstration documented that the Ohio Programs led to large in- 
creases in enrollment, credits earned, and degree receipt. By the end of Year 2, for example, 19 
percent of students in the program group had earned degrees, compared with only 8 percent of 
students in the control group.' This chapter updates those findings through three years, examining 
whether the large effects on degree receipt held up for another year and whether the program led 
to an increase in transfers to four-year colleges. 


Summary of Effects 


The Ohio Programs nearly doubled degree receipt through Year 3. By the end of Semester 6, 35 
percent of the students in the program group had earned degrees, compared with 19 percent of 
students in the control group, a 16 percentage point increase. The programs also increased trans- 
fers to four-year colleges by Semesters 5 and 6. The programs led to positive effects on full-time 
enrollment and credits earned in most of the follow-up semesters. Finally, effects were similar for 
students with and without developmental education requirements at study entry, indicating that 
the Ohio Programs worked equally well for students with different levels of academic prepara- 
tion. In general, the effects were large, positive, and statistically significant across different types 
of students, including students from different racial backgrounds and traditional and nontradi- 
tional students. 


Effects on Degree Receipt and Transfers to Four-Year Colleges 
e The Ohio Programs nearly doubled degree receipt through Year 3. 


Table 5.1 presents data on degree receipt and transfers to four-year colleges. As shown 
in the first panel, by Semester 4, only 8 percent of students in the control group had earned 
degrees. In contrast, 19 percent of students in the program group had earned degrees, an in- 
crease of 11 percentage points. This effect was shown in the earlier brief. Degree receipt con- 
tinued to climb over the semesters. By Semester 6, 19 percent of the control group had earned 
degrees compared with 35 percent of the program group, for an estimated effect of about 16 
percentage points.” 


'Sommo, Cullinan, and Manno (2018). 


Put another way, this impact indicates that the Ohio Programs caused an estimated 126 program group 
students (15.6 percent of the 806 students in the program group) to graduate in three years who would not have 
done so in the absence of the program. 
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Table 5.1 


Degrees Earned and Transfers to Any Four-Year Colleges After Three Years 


Program Control 


Outcome Group Group Difference P-Value 
Earned a degree from any college 
Semester 1 0.3 0.0 0.3* 0.095 
Semester 2 1.6 0.7 0.9 0.116 
Semester 3 7.7 2.5 5.1*** 0.000 
Semester 4 19.0 7.9 11.1*** 0.000 
Semester 5 27.8 12.9 14.9*** 0.000 
Semester 6 34.8 19.2 15.6*** 0.000 
Highest degree earned 
Certificate 1.4 1.7 -0.2 0.728 
Associate's degree 33.2 17.2 16.1*** 0.000 
Bachelor's degree or higher 0.2 0.2 -0.1 0.686 
Registered at a 4-year college 
Semester 1 0.6 0.5 0.1 0.749 
Semester 2 1.9 2.9 -1.0 0.243 
Semester 3 4.1 4.7 -0.6 0.560 
Semester 4 7.9 7.8 0.1 0.928 
Semester 5 13.6 10.6 2.9% 0.088 
Semester 6 17.5 11.8 5.7*** 0.002 
Sample size (total = 1,501) 806 695 


SOURCE: MDRC calculations using data from the National Student Clearinghouse and transcript data 
from the study colleges. 


NOTES: Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Degrees earned in spring and summer semesters are combined. 

A two-tailed t-test was applied to differences between research groups. 

Estimates are adjusted by site, cohort, gender, race/ethnicity, age, parental status, marital status, weekly 
hours worked, dependence on parents for 50 percent or more of financial support, whether a student is the 
first family member to attend college, whether a student earned a high school diploma, the number of de- 
velopmental education requirements at the time of random assignment, and intended enrollment level. 


The next panel of the table presents data on the type of degree earned. As expected, all of 
the Ohio Programs’ effect on degree receipt is for associate’s degrees. Few students earned cer- 
tificates, and even fewer earned bachelor’s degrees or higher degrees by the end of Year 3. 


e The Ohio Programs increased transfers to four-year colleges. 


The bottom panel of Table 5.1 presents registration at four-year colleges using data ob- 
tained from the National Student Clearinghouse, which covers most colleges in the United States. 
As might be anticipated, few students in the study were registered at four-year colleges during 
the first four semesters. Transfers to four-year colleges increased in Year 3, and the program in- 
creased registration at four-year colleges by 6 percentage points by Semester 6. 


The effect on transfers to four-year colleges in Semester 6 (6 percentage points) is about 
a third of the size of the effect on associate’s degree receipt in Semester 5 (16 percentage points, 
indicating that the majority of those who earned degrees because of the program did not go on 
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to four-year institutions.’ This low transfer rate is perhaps not surprising, given the barriers to 
college entry low-income students face. National data indicate that only about 33 percent of 
community college students transfer to four-year institutions within six years, even though most 
of them enter college intending to pursue bachelor’s degrees (as was the case in this study 
sample).4 


Effects on Progression Through College 


How did the Ohio programs lead to increased degree receipt and transfers to four-year col- 
leges? This section examines effects on enrollment, credits earned, and developmental re- 
quirements met. 


e The Ohio programs increased enrollment, especially full-time enrollment, 
through Semester 5. 


Figure 5.1 presents data on college enrollment through Semester 6. The figure shows any 
enrollment and full-time enrollment, since students in the program group were required to enroll 
full time. The figure shows that the program group had consistently higher enrollment rates in 
each of the first four semesters. In Semester 3, for example, 68 percent of program group students 
were enrolled at one of the participating colleges compared with 58 percent of control group stu- 
dents, an increase of 10 percentage points. The effect on full-time enrollment was even larger, at 
19 percentage points. 


The effects on enrollment are encouraging and indicate that the support provided by the 
programs helped students stay in school when they otherwise would have dropped out. The 
positive effects on full-time enrollment are due to a combination of this support and the full- 
time requirement. Note that nearly all program group students who were enrolled in Semester 
1 were enrolled full time, as required by the program. By Semester 4, however, only about 60 
percent of enrolled students were enrolled full time, illustrating a fall in program participation 
over time. Program group students who were enrolled but were not enrolled full time were not 
eligible for the program’s financial support but were eligible for advising, tutoring, and career 
services. 


Enrollment dropped steadily over the semesters, as students either earned degrees or left 
school. The effects on enrollment similarly decline and are small and statistically insignificant by 
Semester 6. 


>For students whose fifth semester after joining the study was a fall semester, it may have been difficult to 
register at a four-year college the next semester, since it was a spring semester. However, this general pattern 
was also found among only those students whose fifth semester was a spring semester. 


4Ma and Baum (2016). 
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Figure 5.1 


The Ohio Programs Boosted Enrollment 
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SOURCE: MDRC calculations using data from the National Student Clearinghouse and transcript data from the study colleges. 


NOTES: Enrollment is based on all available data and combines spring and summer enrollment. 

FT = full-time, defined as enrollment in 12 or more credits and based on data from the college of random assignment only. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 
percent. 

Estimates are adjusted by site, cohort, gender, race/ethnicity, age, parental status, marital status, weekly hours worked, dependence on parents for 50 
percent or more of financial support, whether a student is the first family member to attend college, whether a student earned a high school diploma, the number 
of developmental education requirements at the time of random assignment, and intended enrollment level. 


e The Ohio programs increased college-level credits earned during each of 
the first five semesters and in total. 


Table 5.2 presents data on credits earned, overall and by type. The top panel presents 
total credits (developmental and college-level) and the next two panels present credits by type.° 
The Ohio Programs led to an increase of just over 2 credits in each of the first three semesters 


Table 5.2 
Credits Earned After Three Years 


Program Control 


Outcome Group Group Difference P-Value 

Total credits earned 
Semester 1 10.1 8.1 21°" 0.000 
Semester 2 8.5 6.1 2.4*** 0.000 
Semester 3 6.7 4.3 230°" 0.000 
Semester 4 5.2 3.8 1:5°** 0.000 
Semester 5 3.0 2.6 0.5* 0.070 
Semester 6 2.1 2.2 -0.1 0.816 
Cumulative total credits earned 34.9 26.4 8.5*** 0.000 

Developmental credits earned 
Semester 1 1.9 1.6 0.2* 0.052 
Semester 2 0.7 0.6 0.2** 0.048 
Semester 3 0.2 0.2 0.0 0.513 
Semester 4 0.1 0.1 0.0 0.704 
Semester 5 0.1 0.1 0.0 0.887 
Semester 6 0.0 0.0 0.0 0.631 
Cumulative developmental credits earned 2.8 2.5 0.2 0.177 


College-level credits earned 


Semester 1 8.3 6.4 1.8*** 0.000 
Semester 2 7.8 5.6 2.2*** 0.000 
Semester 3 6.5 4.1 2.4*** 0.000 
Semester 4 5.1 3.6 1.5°** 0.000 
Semester 5 2.9 2.5 0.5* 0.062 
Semester 6 2.1 2.1 0.0 0.845 
Cumulative college-level credits earned 32.1 23.8 8.2*** 0.000 
Sample size (total = 1,501) 806 695 


SOURCE: MDRC calculations using transcript data from the study colleges. 


NOTES: Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Credits earned in spring and summer semesters are combined. 

A two-tailed t-test was applied to differences between research groups. 

Estimates are adjusted by site, cohort, gender, race/ethnicity, age, parental status, marital status, 
weekly hours worked, dependence on parents for 50 percent or more of financial support, whether a 
student is the first family member to attend college, whether a student earned a high school diploma, 
the number of developmental education requirements at the time of random assignment, and intended 
enrollment level. 


Students do not earn college-level credits for developmental courses, meaning that those credits do not 
count toward degree requirements. 
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and 1.5 credits in Semester 4. As was seen for enrollment, the effects diminish over the semes- 
ters and are statistically insignificant by Semester 6. In total, students in the program group 
earned 8.5 more credits over the follow-up period than their counterparts in the control group. 


Note that credits earned reflect those earned during the three-year follow-up period, and 
many students entered the study with some credits already earned. Students were eligible for the 
study if they had accumulated 24 or fewer credits, and on average, students entered the study with 
about 10 credits completed. One-third of students had no credits at study entry. 


e Although they did not affect developmental credits earned, the Ohio Pro- 
grams did increase the number of students who met their developmental 
education requirements. 


About 75 percent of students entered the study with at least one developmental education 
requirement: a developmental course that they were required to take. The programs encouraged 
students to take these courses early. Nonetheless, students earned few average credits in develop- 
mental education courses (indicating that few students took these courses), and the program’s 
effect on this outcome was small. 


Nevertheless, the program did increase the number of students who completed their de- 
velopmental education requirements, as shown in Table 5.3. By Semester 6, for example, 49 per- 
cent of students in the program group had met these requirements compared with 37 percent of 
students in the control group, an effect of 12 percentage points. There could be a positive effect 


Table 5.3 


Completion of Developmental Requirements After Three Years 


Program Control 


Outcome Group Group Difference P-Value 

Completed developmental requirements 
Semester 1 20.8 14.5 6.3*** 0.004 
Semester 2 33.6 22.6 ATA 0.000 
Semester 3 42.2 27.8 14.4*** 0.000 
Semester 4 45.2 33.0 12.2*** 0.000 
Semester 5 47.3 35.1 12.1°** 0.000 
Semester 6 49.1 36.8 12.2*** 0.000 

Sample size (total = 1,060) 567 493 


SOURCE: MDRC calculations using data from the National Student Clearinghouse and transcript data 
from the study colleges. 


NOTES: Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Degrees earned in spring and summer semesters are combined. 

A two-tailed t-test was applied to differences between research groups. 

Estimates are adjusted by site, cohort, gender, race/ethnicity, age, parental status, marital status, 
weekly hours worked, dependence on parents for 50 percent or more of financial support, whether a stu- 
dent is the first family member to attend college, whether a student earned a high school diploma, the 
number of developmental education requirements at the time of random assignment, and intended enroll- 
ment level. 
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on this outcome even though the programs had little effect on developmental credits earned be- 
cause students can meet developmental requirements in several ways: They can take and pass 
developmental education courses, but they can also retake their placement tests and place into 
college-level courses, or they can pass college-level courses in the relevant subjects. Data from 
the colleges indicate that most students with developmental education requirements at study entry 
met these requirements through the latter two methods. 


Effects for Subgroups 


In addition to examining the overall average effect of the Ohio Programs, the study also assessed 
whether the program was effective for various types of students. The primary and confirmatory 
subgroup analysis (the analysis that will provide rigorous evidence on the central question for this 
section) focused on whether effects differed for students with developmental requirements com- 
pared with those without these requirements, since the former group is most at risk of dropping 
out of college. In addition, the original evaluation of the City University of New York (CUNY) 
Accelerated Study in Associate Programs (ASAP) included only students with developmental 
requirements, so there is interest in experimentally assessing whether the program can work as 
well for students without these needs.° 


This study also examined effects across other subgroups, including groups defined by 
gender, race, college attended, age, high school diploma status, nontraditional student status, and 
semester of joining the study. Because there are many subgroups being tested, the likelihood in- 
creases of finding one or two significant differences simply by chance. Thus these analyses are 
exploratory, meaning that any positive findings must be viewed as suggestive only. Because im- 
pacts for subgroups are expected to vary to some extent simply by chance, the main focus of the 
subgroup analysis is not on the effect for a given subgroup, but on the difference in effects be- 
tween the two subgroups and whether that difference is statistically significant. 


e The Ohio programs increased degree receipt and credits earned for stu- 
dents with and without developmental education needs. 


Table 5.4 present effects for students with and without developmental requirements. The 
table indicates that the program had very similar estimated effects for both types of students. The 
effects themselves are of similar size and the differences between them are not statistically sig- 
nificant. This result is encouraging for two reasons. First, the finding replicates the CUNY ASAP 
findings showing that the model works for less academically prepared students. The lower per- 
formance of these students is evident in the table: Only 16 percent of the students in the control 
group with developmental requirements earned degrees during the period, compared with 28 


Although the MDRC evaluation of CUNY ASAP only included students with developmental education 
needs, CUNY’s quasi-experimental evaluation of the program included students with and without developmental 
education requirements and found positive effects for both types of students. See Strumbos and Kolenovic 
(2017). 
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Table 5.4 


Total Credits Earned and Total Degrees Earned After Three Years: 
Variation in Effects by Developmental Education Requirements 


P-Value for 
P-Value Differential 
Sample Program Control for Estimated 
Outcome Size Group Group Difference Difference Effects 
Cumulative credits earned 0.8090 
With developmental education requirements 1,060 34.3 25.2 9.1 *** 0.0000 
Without developmental education requirements 366 38.5 30.2 8.3 *** 0.0020 
Sample size 1,426 
Degrees earned (%) 0.8220 
With developmental education requirements 1,060 32.0 16.3 15.7 *** 0.0000 
Without developmental education requirements 366 45.0 28.0 17.0 *** 0.0010 
Sample size 1,426 


SOURCE: MDRC calculations using data from the National Student Clearinghouse, placement test data, and transcript data 
from the study colleges. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are indicated as: 
*** = 1 percent; ** = 5 percent; * = 10 percent. 

A two-tailed t-test was applied to differences in effects between subgroups. Statistical significance levels are indicated as: 
ttt = 1 percent; tt = 5 percent; + = 10 percent. For the measures presented in this table, no statistically significant 
differences in effects between subgroups were observed. 

Estimates are adjusted by site, cohort, gender, race/ethnicity, age, parental status, marital status, weekly hours worked, 
dependence on parents for 50 percent or more of financial support, whether a student is the first family member to attend 
college, whether a student earned a high school diploma, and intended enrollment level. 


percent of those without developmental requirements. Among those students with developmental 
requirements, the programs had a 16 percentage point impact. Second, the finding indicates that 
the program can also benefit more academically prepared students. This experimental finding for 
the effectiveness of the ASAP model confirms previous quasi-experimental evidence that these 
students can benefit from the additional support. 


e The Ohio Programs generally had positive effects for various types of stu- 
dents. The programs appear to have had larger effects for women than 
for men. 


Appendix Tables E.1 and E.2 presents effects on credits earned and degree receipt for the 
additional subgroups. Across all subgroups, the estimated effects of the Ohio Programs are gen- 
erally large and positive, showing that the program worked equally well for many types of stu- 
dents. The one exception is for subgroups defined by gender. The programs’ effects are large and 
positive for men, but they are even larger for women. For example, the Ohio Programs increased 
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degree receipt for women by 18 percentage points compared with 10 percentage points for men. 
The difference between these subgroup impacts is statistically significant. This finding should be 
interpreted with caution, given the number of subgroups tested and the fact that different effects 
by gender were not observed in the evaluation of CUNY ASAP in New York. Nonetheless, it is 
an interesting pattern and warrants further research and attempted replication.’ 


Conclusion 


The Ohio Programs increased enrollment in each of the first five semesters, increased full-time 
enrollment, and led to an increase in total credits earned. As a result, more students in the program 
group earned degrees by the end of Year 3 and more transferred to four-year colleges. 


’The subgroup analysis also indicates that the program’s effects did not differ for students of different races 
or ethnicities. This pattern contrasts with recent findings from CUNY ASAP, showing very small effects on six- 
year degree receipt for Hispanic students compared with positive effects for black and white students. See Weiss 
et al. (2019). 
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Chapter 6 


The Costs of the Ohio Programs 


This chapter analyzes the resources that the colleges invested in the Ohio Programs in relation to 
the programs’ estimated effects. First, it identifies the costs of the programs, including the costs 
of the components associated with their comprehensive services. Next, it compares the total cost 
of college for program group students with the total cost for students receiving the usual college 
services. Last, it compares the total cost of college with students’ academic outcomes for both 
groups in order to explore how the programs changed the cost per outcome achieved — specifi- 
cally, whether the investment in the programs produced more graduates per dollar spent within 
three years than the usual college services. 


Summary of Findings 


The main findings are as follows: 


e The direct cost of the program services is $5,521 per program group member 
over three years, or $1,840 per program group member per year. The three- 
year estimate includes $2,369 for administration and staffing, $1,517 for stu- 
dent services, and $1,635 for financial support. 


e When the additional costs of educating students are considered, over the three 
years of the follow-up period the college invested $8,030 more per program 
group member than it did per control group member. This estimate includes 
the direct cost to operate the programs ($5,521) plus an estimate of the cost 
associated with program group students attempting more college courses dur- 
ing that time ($2,510). This additional $2,510 is offset by additional revenue 
generated by these students. 


e A cost-effectiveness analysis shows that the programs lowered the cost per 
degree earned within three years. The additional $8,030 investment (a 42 per- 
cent increase) in each program group student led to a large increase in degree 
receipt, with a cost of approximately $51,000 per additional degree, consider- 
ably less than the cost per graduate in the control group of $99,000. Relatedly, 
the cost per degree earned for program group students was 22 percent less than 
the cost per degree for control group students. 


Calculating Direct Costs 


The direct costs of the Ohio Programs are those incurred for administration and staffing, student 
services, and financial support. Table 6.1 shows the total annual direct cost per program group 
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Table 6.1 


Direct Costs of the Programs per Sample Member 


Total Three-Year Percentage of the 
Program Component Cost per Year ($) Cost ($) Total 


Administration and staffing 


Administration 704 2,112 38.3 
Institutional research 29 86 1.6 
Other 57 171 3.1 
Subtotal 790 2,369 42.9 


Student services 


Advising 415 1,244 22.5 
Career services 55 166 3.0 
Tutoring 36 107 1.9 
Subtotal 506 1,517 27.5 


Financial support 


Monthly incentive 160 479 8.7 
Textbook assistance 210 631 11.4 
Tuition waiver 175 525 9.5 
Subtotal 545 1,635 29.6 


Total 1,840 5,521 100.0 


SOURCE: MDRC calculations based on program-expenditure data from the study colleges from 
October 2014 through December 2018. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 

Program costs are based on total costs during the first five years of the program, including the 
pilot phase. 

The discount rate used for program costs is 3 percent. All costs are shown in constant 2018 
dollars. 


student: $1,840, or a total of $5,521 over three years. This estimate includes all students who 
were offered the Ohio Programs, including those who enrolled less than full time, dropped out, 
or graduated. Cost results are described using this approach (rather than a cost-per-full-time- 
equivalent approach) in order to align these cost estimates with the outcomes and effects de- 
scribed in Chapter 5, which also include all students in the program and control groups. In 
addition, these costs reflect an estimate of the expected additional funding required per student 
offered the Ohio Programs. Built into that expectation is the understanding that some students 
will stop or drop out. 


Direct costs per student were calculated by taking the total cost of the programs and di- 
viding it by the number of students assigned to the program group. To calculate the direct cost 
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per student per year, the direct cost per student was then divided by the average number of aca- 
demic years since those students were offered the programs (approximately three).! 


Definitions of Direct Cost Categories 


The direct costs of the Ohio Programs consist of three main components: administration 
and staffing, student services, and financial support. This section describes the types of activities 
and expenditures included in each of these components and how costs were allocated across 
them.” 


Administration and staffing costs consist of: 


e Administration — the salaries, benefits, and overhead associated with senior 
advisers, the program director, and staff associates at the individual colleges 
who manage the programs? 


e Institutional research — costs associated with colleges’ internal data collec- 
tion and analysis of the programs (not including costs associated with MDRC 
researchers’ evaluation of the programs) 


e Other — office supplies, consultants, travel, marketing materials, computers 
Student services costs consist of: 
e Advising — salaries, benefits, and overhead for program advisers 


e Career services — salaries, benefits, and overhead for program-specific ca- 
reer-services staff members 


e Tutoring — salaries, benefits, and overhead for program-specific tutors for 
the proportion of their time they spent working with program group students 
enrolled in developmental courses, on academic probation, or otherwise seek- 
ing assistance 


Financial support costs consist of: 


e Tuition waiver — the dollar amounts of any differences between financial aid 
and tuition and fees (which are waived as part of the programs) 


'The “number of students assigned to the program group” includes 533 students who were not part of the 
impact analysis sample but who did have program services made available to them. The time period is approxi- 
mately 3 years because the number of students in each cohort varies, making the number of weighted average of 
years since beginning the programs 2.9. 

2 Appendix F provides additional cost calculations, including alternate calculations of direct costs at a steady 
state and costs for enrollees (excluding members of the program group who did not enroll). All cost estimates 
are based on data through the end of 2018. 

3Overhead refers to costs that are not direct labor or direct materials costs, for example costs for utilities or 
furniture. 
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e Textbook assistance — voucher amounts to cover students’ textbook costs 
through the campus bookstore 


e Monthly incentive — monthly $50 gas/grocery gift cards given to students 
contingent on their participation 


About 43 percent of the direct cost of the programs comes from administration and staff- 
ing, mostly from senior leaders and the fully dedicated program directors who managed the pro- 
grams and provided quality control. 


Financial support — including the monthly incentives, textbook subsidies, and tuition 
assistance provided to program group students — makes up 30 percent of the programs’ cost. 
Textbooks are the biggest expense in this category, accounting for just over a third of the 
costs, followed by the tuition waivers. Monthly incentives averaged $160 per program group 
student per year (the average is calculated including students who did not enroll or complete 
all requirements). 


Finally, about 28 percent of the direct cost of the programs comes from the student ser- 
vices provided, mostly from the fully dedicated advisers. Tutoring and career-services costs are 
quite minimal because the colleges were able to use their existing resources in these areas. 


Calculating Base Costs 


Table 6.2 shows the programs’ direct cost, base cost, indirect (or induced) cost, and net cost. 
“Base cost” refers to the cost of the “usual” college services provided to students who are not in 
the programs — the cost of instructors, buildings, college administration, etc. The base cost pro- 
vides context for interpreting the programs’ direct cost. 


This base cost estimate assumes that resource use corresponds to the number of credits 
attempted; in other words, a student who attempts more credits is generally associated with greater 
expenditures than a student who attempts fewer credits. The analysis uses credits attempted be- 
cause it provides a simple measure of a student’s level of engagement with the college. To esti- 
mate the dollar value of the credits attempted in a usual college experience, the number of credits 
attempted per year by students in the control group (about 11.4 from random assignment through 
the end of 2018) is multiplied by an estimated cost per credit. This total comes to about $6,350 in 
college operating costs per year, $19,039 total, for each student in the control group.‘ 


‘The Integrated Postsecondary Education Data System (IPEDS) of the National Center for Education Sta- 
tistics provides data on college expenditures and instructional activity credit hours. MDRC calculated costs per 
credit by taking the IPEDS total expenses for the participating colleges (including developmental course costs) 
and dividing by the IPEDS instructional activity credit hours, a number which — unfortunately for this purpose 
— does not include developmental credits. Because developmental courses are included in the numerator but not 
the denominator, costs per credit using IPEDS data are likely to be overestimated. See Romano et al. (2019). 
However, this overestimate occurs for both groups’ base costs, which would cancel out in the net cost calculation. 
Indirect or induced costs may still be overestimated. 
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Table 6.2 
Net Costs of Education per Sample Member Over Three Years 


Program Control Difference 
Feature ($) Group Group (Net) 


Direct cost: cost of primary program components 5,521 0 5,521 
Base cost: cost of credits attempted in the absence of the program 19,039 19,039 0 


Indirect cost: cost of additional credits attempted due to the program 


Upper bound: marginal cost equal to average cost* 5,019 0 5,019 

Lower bound: marginal cost equal to zero” 0 0 0 

Average of upper and lower bound: primary estimate of marginal cost 2,510 0 2,510 
Total cost 

Upper bound: marginal cost equal to average cost* 29,579 19,039 10,540 

Lower bound: marginal cost equal to zero” 24,560 19,039 5,521 

Average of upper and lower bound: primary estimate of total cost 27,069 19,039 8,030 


SOURCE: MDRC calculations using expenditure and transcript data from the study colleges and financial and 
enrollment data from the Integrated Postsecondary Education Data System. 


NOTES: Tests of statistical significance were not performed. 

Rounding may cause slight discrepancies in sums and differences. 

Program costs exclude external research costs. 

Credits attempted include all college-level and developmental credits attempted. 

“Marginal cost equal to average cost” represents the case in which existing college resources cannot be 
used to accommodate changes in credits attempted, so the college incurs additional costs. The additional cost 
to the college, or the marginal cost of the additional credits attempted, is approximated as the average cost per 
credit attempted at the institution, excluding the cost of academic support and student services that the Ohio 
Programs are already providing. 

>“Marginal cost equal to zero” reflects the case in which the college can absorb the cost of additional credits 
attempted by the program group using existing resources and without incurring new costs. 


This base cost is an estimate of how much money is spent to educate the typical student 
in the absence of the programs. One limitation of this approach is the assumption that all credits 
attempted have the same cost to the college, which is probably not the case. For example, science 
lab courses may be more expensive than English courses. The analyses also assume that the av- 
erage cost of serving a student at the college is similar to the average cost of serving a student in 
the study sample. 


Calculating Indirect Costs 


There are additional costs to a college if students take more credits because of a program, as they 
have done in this study. These are referred to as indirect, or induced, costs. If a small number of 
program group students take additional courses, the college would probably incur no marginal 
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costs. However, at some point rising enrollment would lead to increased costs, as colleges would 
need to add more courses and hire more staff members. 


Indirect costs are estimated based on the average number of additional credits attempted 
by program group students compared with control group students. This analysis presents three 
estimates. A lower-bound estimate assumes that the indirect costs equal zero — that is, that the 
college incurs no additional costs when more students enroll or when students attempt additional 
credits. An upper-bound estimate is based on average costs, excluding the costs of academic ser- 
vices and student services. The upper-bound estimate represents the case where the college is 
unable to absorb the cost of additional students enrolled or additional credits attempted because 
its existing resources are already fully used. For example, if students are enrolling in additional 
courses that are filled to capacity, then the college may have to open new course sections. 


It is unlikely that every additional credit attempted by a student costs the college as much 
as the average credit attempted, and it is also unlikely that there is zero cost to the college for 
additional credits attempted. An average of these two estimates — the midpoint between the up- 
per and lower bounds — is therefore used as the primary estimate of indirect costs. That midpoint 
is $837 of indirect costs per student per year, $2,510 total over three years. This estimate is in- 
tended to approximate the indirect costs should these programs continue, and to provide a useful 
estimate to other colleges. However, for the time period covered in this report, the colleges in this 
study were facing underenrollment challenges, so the true indirect cost may have been closer to 
the lower bound of $0. 


Calculating Total and Net Costs 


The costs of each group are presented in the final lines of Table 6.2. The total cost is calculated 
by adding the direct cost, base cost, and indirect cost. The total cost over three years per program 
group member is $27,069, compared with $19,039 per control group member. The net cost is 
defined as the difference between the total program group cost and the total control group cost. 
This net cost is $8,030 per program group member, representing a 42 percent increase over the 
base cost. 


Sustainability Considerations 


If program group students took additional credits, the colleges should have received additional 
tuition, and that increased revenue should offset the indirect costs somewhat. State funding for 
improved performance should also provide the colleges some additional revenue. Using tuition 
data from IPEDS and estimates of the programs’ effects on credits attempted (calculated as part 
of the evaluation but not shown in Chapter 5), it is possible to determine that the colleges should 
have received about $400 in additional tuition per program group student per year as a result of 
the programs. Additionally, the state of Ohio awards funding to colleges per degree, certificate, 
and credit students earn; per student who transfers to a four-year college; and per student who 
reaches certain credit benchmarks (12, 24, 36). It provides additional money when at-risk students 
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reach these milestones.* These funding formulas should have brought the colleges at least another 
$750 per student per year. Under these assumptions, the total revenue increase from tuition and 
performance-based funding due to the programs over three years is about $3,450, or nearly half 
of the net cost calculated above. In other words, although the estimated net cost of the program is 
about $8,000 over three years, the estimated net revenue is about $3,450 — so the cost minus 
revenue is around $4,500 per student over three years. 


Cost-Effectiveness Analysis 


A cost-effectiveness analysis expresses the costs of alternative interventions as costs per unit of a 
desired outcome. This cost-effectiveness analysis considers the cost per degree earned within 
three years and the cost per credit earned. These estimates spread costs across all students who 
were offered the Ohio Programs, including those who enrolled less than full time, dropped out, 
or graduated. 


Table 6.3 shows the cost-effectiveness calculations for the programs over three years. 
The first row shows the total and net costs from Table 6.2. The next row shows the average per- 
centage of students who completed degrees in three years: 35 percent of the program group and 


Table 6.3 


Cost-Effectiveness Values 


Program Control Difference 
Outcome Group Group (Impact) 
Cost per group member over three years ($) 27,069 19,039 8,030 
Earned a degree (%) 34.8 19.2 15.6 *** 
Cost per degree earned ($) 77,783 99,162 -21,378 
Total credits earned 34.9 26.4 8.5 *** 
Cost per credit earned ($) 776 721 54 
Sample size (total = 1,501) 806 695 


SOURCE: MDRC calculations using program-specific participation and budget data, transcript data, and 
financial and enrollment data from the Integrated Postsecondary Education Data System. 


NOTES: Rounding may cause slight discrepancies in sums and differences. All dollar values have been 
rounded to the nearest whole dollar. 

Tests of statistical significance have only been performed on outcome measures, not costs. All outcomes 
are cumulative over three years. For these measures, a two-tailed t-test was applied to differences between 
research groups. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by site, cohort, gender, race/ethnicity, age, parental status, weekly hours worked, 
dependence on parents for 50 percent or more of financial support, whether a student is the first family 
member to attend college, whether a student earned a high school diploma, the number of developmental 
education requirements at the time of random assignment, and intended enrollment level. 


SOhio Department of Higher Education (2017). At-risk students (called “access students” in Ohio) are de- 
fined based on age, race, financial need, and academic preparation. 
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19 percent of the control group (from Table 5.1). The average cost per degree for each group is 
the cost per group member divided by the percentage who received degrees. As shown in the 
table, the cost per degree earned is over $20,000 less for the program group. The programs are 
more cost-effective for this outcome than the status quo. 


Continuing down Table 6.3, the next line shows the average credits earned in three years 
for each group (from Table 5.2). The average cost per credit earned for each group is calculated 
by dividing the cost per group member by the average credits earned, resulting in a slightly higher 
cost per credit earned for the program group than the control group. 


The programs caused 126 students to graduate within three years who would not have 
otherwise. They had a total additional cost of $8,030 in net cost per program group student. This 
net cost per student multiplied by 806 students in the program group equals roughly $6.47 million. 
Dividing the $6.47 million by 126 provides the cost per additional graduate: $51,000, considera- 
bly less than the cost per graduate in the control group of $99,000. Using a similar calculation, 
the incremental cost per credit earned (the cost per credit earned for those additional credits 
earned because of the program) is $945, which is higher than either group’s average cost per 
credit earned. Thus, for the outcome of credits earned, the programs are not more cost-effective 
than the status quo. 


Taken together, these findings suggest that the Ohio Programs are cost-effective at get- 
ting students to take and pass a relatively small number of additional courses they need in order 
to fulfill degree requirements, but not cost-effective at getting them to take and pass large num- 
bers of additional courses. Since a degree is often considered the primary purpose of taking 
credits, cost-effectiveness per degree is likely to be the more relevant of the two outcomes con- 
sidered here. 
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Chapter 7 


How Do the Ohio Programs Compare 
with the Original Accelerated Study in Associate 
Programs (ASAP)? 


A central question in the evaluation of the Ohio Programs is whether the ASAP model could be 
implemented in a different context from the City University of New York (CUNY) and achieve 
similar results. This chapter presents a comparison of the contexts and findings from both studies.! 
It first compares the program models in the two locations, their implementation, and the types of 
students they served. It then compares their effects on academic outcomes and estimates of costs 
and cost-effectiveness. 


The Ohio Programs’ Changes to the CUNY ASAP Model 


While the Ohio Programs are closely based on CUNY ASAP, some changes were made to the 
program model for its application in Ohio. These changes were made to accommodate the new 
community college structures and a new geographic location. 


First, the Ohio colleges’ course structure differed from CUNY’s. CUNY offered winter 
enrollment — that is, an opportunity to earn credits between the fall and spring terms — and the 
Ohio colleges did not. Therefore, Ohio programs could not encourage winter session enrollment 
as a method to accelerate students’ progress. CUNY also offered program group students an 
ASAP-only student success seminar in their first year. Ohio colleges did not create a special 
course for program group students. Instead they enrolled program group students into specific 
sections of existing student success courses. 


Second, CUNY ASAP program management — including oversight, data collection, re- 
porting, and iterative improvement — was largely handled centrally by a dedicated ASAP team 
inthe CUNY Office of Academic Affairs, in partnership with CUNY colleges. The Ohio colleges 
had decentralized governance, with local boards for each college. Therefore, the Ohio Programs 
implemented a local data-collection and management structure, with college leaders providing 
oversight. Furthermore, CUNY’s ASAP includes dedicated career and employment specialists 
that the Ohio Programs did not; the Ohio Programs instead chose to connect program group stu- 
dents to the colleges’ existing career services. 


Finally, CUNY provided an incentive of a monthly, unlimited-ride MetroCard for meet- 
ing participation requirements. That incentive was hard to match in Ohio given the lack of public 
transportation. Instead, the Ohio colleges offered program group students $50 gift cards. 


'The findings from the original CUNY ASAP evaluation described in this chapter are from Scrivener et al. 
(2015). 
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The ASAP model is not static, and CUNY continues to refine it. A one-to-one compari- 
son of MDRC’s ASAP evaluation findings with the Ohio findings is therefore not completely 
straightforward. For example, one significant change CUNY has made since the original ASAP 
evaluation has been to its advising: Originally, ASAP required all students to meet with an adviser 
twice per month regardless of their needs. As CUNY expanded ASAP to serve more students, it 
began to sort students in their second semester and beyond into low-, medium-, and high-need 
groups, with different advising requirements for students in each group. At the same time CUNY 
adjusted the adviser/student ratio upward from 60 to 80 students per adviser (as it was during the 
original evaluation period) to 150 students per adviser. The Ohio Programs were modeled on this 
adjusted version of advising.” As a result, students in the original evaluation should be expected 
to have spoken with advisers more often in their first year than students in the Ohio Programs. 


CUNY also implements some ASAP components that were not part of the Ohio Pro- 
grams model.* For example, the ASAP Summer Institute is a one-day experience designed to 
familiarize students with essential college and ASAP policies, build the ASAP community 
through team-building activities, and allow students to meet staff members and each other be- 
fore the start of the semester. Also, the ASAP Student Leader Program, administered by the 
ASAP Central Office, is designed to provide opportunities for current ASAP students to explore 
and practice leadership skills and to expand the recruitment capabilities of individual ASAP 
partner colleges. 


Participating Students 


The students in the Ohio evaluation differed from those in the CUNY evaluation in various 
ways. For example, more Ohio sample students were nontraditional, meaning they were older, 
were working full time, or had children. This difference in the sample is notable, as there was 
some concern that the model, with its full-time attendance requirement, would not work for 
these types of students. Fewer Ohio sample students had developmental requirements than was 
the case in the CUNY evaluation.* Ohio students were also closer to obtaining degrees when 
they entered the study than CUNY students, meaning that they entered the study with more 
credits already earned. 


Students’ Experiences and Service Contrast 


The Ohio Programs evaluation finds that the service contrasts between the program group and 
the control group, although large, were somewhat smaller than in the original CUNY ASAP eval- 
uation. The differences in receipt of advising, tutoring, and career services are statistically signif- 
icant in both studies, but the differences were smaller in Ohio than they were in New York. The 


2See Sommo and Ratledge (2016). 
3Boykin and Prince (2015). 
4The MDRC evaluation of CUNY ASAP was restricted to students with developmental education require- 


ments. CUNY’s quasi-experimental evaluation of ASAP was not restricted to this group. See Strumbos and 
Kolenovic (2017). 
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contrasts were not as large in Ohio because fewer program group students reported meeting with 
an adviser, going to tutoring, or meeting with a career services staff member in the first year than 
was the case among program group students in the CUNY evaluation. A similar pattern emerges 
in the comparison of the number of times students met with an adviser, attended tutoring, or vis- 
ited career services, as shown in Figure 7.1. Furthermore, fewer program group students in Ohio 
experienced any classes with other program group students (through the Ohio Programs’ efforts 
to mimic block scheduling) than did CUNY program group students. Together, these differences 
could suggest some potential diminishing marginal returns to these services, since the Ohio Pro- 
grams achieved relatively similar effects on academic outcomes with smaller service contrasts. 


On the other hand, there are some indications that the Ohio control group received a bit 
more advising support than students in New York, which may reflect the natural development of 
student support practices nationwide. For example, students in the control group in Ohio reported 
meeting with advisers for longer each time, on average, than the control group at CUNY. Also, 
control group students in Ohio discussed a broader range of topics with their advisers than the 
control group in New York. 


Finally, there are indications that the college funding situation in New York between 
2010 and 2013 was very different from that in Ohio between 2015 and 2018.° According to the 
student survey data in Ohio, students in both the program and control groups had a harder time 
covering their school expenses than CUNY students did. More Ohio students than CUNY stu- 
dents reported choosing not to register for college full time because they could not afford it 
(though the differences between the program and control groups in each study are similar). More 
students in Ohio than at CUNY said that they needed student loans and employment earnings to 
pay for college. 


Furthermore, more students in Ohio than at CUNY reported having jobs in their first year 
of school and said that they had to miss class often or very often due to work. Unlike with CUNY 
students, in Ohio there was a statistically significant difference between the program and control 
groups in how often they said they missed class because they could not pay for transportation. 
Some of the difference can be attributed to the drastic difference in the modes of transportation 
students used in Ohio compared with New York. However, there were also changes in federal 
financial aid calculations or regulations over time, and there are differences in the states’ level of 
financial support coverage. New York State’s Tuition Assistance Program provides more cover- 
age and greater support to a wider group of students than does Ohio’s College Opportunity Grant.° 
Ultimately, these financial challenges did not keep the Ohio Programs from having positive ef- 
fects on academic outcomes. 


5More program group students in Ohio required tuition waivers than did program group students in MDRC’s 
evaluation of CUNY ASAP. 

®New York State Higher Education Services Corporation (2019); Ohio Department of Higher Education 
(2019). 
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Figure 7.1 


The Ohio Programs’ and CUNY ASAP's Effects on Students’ Experiences 
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SOURCE: MDRC calculations using data from the student surveys in the ASAP Ohio Demonstration and the CUNY ASAP evaluation. 


NOTE: Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 


Career services 


Academic Outcomes and Effects 


Despite some differences in characteristics between the two study samples described above, the 
academic trajectories of the students in the two studies were very similar over the six semesters 
of the follow-up period. For example, as shown in Figure 7.2, similar percentages of students had 
received degrees by Semester 6 in both studies (the percentages were also similar in previous 
semesters, not shown). By Semester 6, 19 percent of control group students in Ohio had earned 
degrees, compared with 22 percent of control group students in New York. Similar percentages 
of the two studies’ control groups also transferred to four-year colleges. 


The programs’ effects were also strikingly similar, across a wide range of outcomes. The 
Ohio Programs increased degree receipt by 16 percentage points at the end of Semester 6, com- 
pared with 18 percentage points for CUNY ASAP. Similarly, the programs increased the propor- 
tion of students who transferred to four-year colleges by 6 percentage points in Ohio and by 8 
percentage points in New York. 


Finally, an important question is whether the programs had similar effects for students 
who entered college with different levels of academic preparation. The MDRC CUNY evaluation 
only included students with developmental education requirements and thus was only able to 


Figure 7.2 


Graduation and Four-Year College Transfer Rates After Three Years 
for the Ohio Programs and CUNY ASAP 


Percentage of students 


Ohio Programs CUNY ASAP Ohio Programs CUNY ASAP 
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SOURCE: MDRC calculations using National Student Clearinghouse data and transcript data from the 
study colleges; Scrivener et al. (2015). 


NOTES: Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

A two-tailed t-test was applied to differences between research groups. 

Degrees earned in spring and summer semesters are combined. 

Estimates for CUNY ASAP are regression-adjusted, controlling for site and cohort. Estimates for 
the Ohio Programs are regression-adjusted, controlling for site, cohort, and a range of student 
characteristics measured at study entry. 
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address this issue by estimating effects for students with different levels of developmental needs. 
CUNY conducted its own quasi-experimental evaluation of ASAP for a broader group of students 
and found positive effects for those with and without developmental education requirements.’ 
The Ohio evaluation, using an experimental comparison, corroborates that finding. The Ohio Pro- 
grams led to similarly large increases in degree receipt for both types of students. 


The Costs of the Programs and Their Cost-Effectiveness 


As noted in the earlier report on the Ohio Programs, almost every category of costs was lower in 
the Ohio Programs evaluation than in the 2010 CUNY ASAP evaluation, probably because 
CUNY had to pay higher New York City salaries, dedicated tutors and career specialists, costs 
associated with blocked and linked courses, and higher monthly incentives in the form of Metro- 
Cards. CUNY ASAP’s costs have dropped substantially over time as the model has evolved and 
has been expanded to serve many more students. 


The Ohio Programs model was based on a modified, less expensive version of CUNY 
ASAP than the one MDRC evaluated. The cost-effectiveness of the Ohio Programs follows a 
very similar pattern to that found in the analysis of CUNY ASAP in New York. In particular, the 
programs increased the cost per credit earned but greatly reduced the cost per degree earned. 


7Strumbos and Kolenovic (2017). 


66 


Chapter 8 


The Implications of the Findings and 
the Future of the Model 


As one of the few programs proven to increase college completion rates dramatically for low- 
income students, the City University of New York (CUNY) Accelerated Study in Associate Pro- 
grams (ASAP) has gained national attention. A vital question for stakeholders in higher education 
was whether the program that originated at CUNY could be effective in other localities or insti- 
tutions. The ASAP Ohio Demonstration shows that it can. Three community colleges in Ohio 
successfully adopted and operated programs based on CUNY ASAP, and the programs achieved 
positive results similar to those of the original program — a remarkable feat. This concluding 
chapter summarizes the evaluation’s findings and discusses their potential implications. It also 
describes the future of ASAP in Ohio and in other localities across the nation. 


Summary of the Findings 


MDRC evaluated the implementation of the three Ohio Programs and their effect on students’ 
academic outcomes. With the exception of blocked schedules, the colleges generally imple- 
mented all program components, and students in the Ohio Programs received significantly more 
advising, tutoring, career services, and financial support than students in the control group. The 
increased support improved student outcomes: The Ohio Programs nearly doubled the percentage 
of students who completed their associate’s degrees in three years, from 19 percent to 35 percent. 
The programs cost about $8,000 more per student than the colleges’ business-as-usual services, 
but the programs increased graduation rates so much that the cost per degree earned was 22 per- 
cent lower for program group students than it was for control group students. 


What These Findings Mean for Higher Education 
Institutions and Policymakers 


Higher education practitioners, policymakers, and other stakeholders continue to search for strat- 
egies to improve the nation’s low community college graduation rates. The positive findings from 
the ASAP Ohio Demonstration point to the following important implications for higher education 
institutions and policymakers. 


e Comprehensive programs that provide multiple years of proactive advis- 
ing, financial support, and other services are proven to have large, posi- 
tive effects on graduation rates. 


This study in Ohio adds to a body of evidence showing that comprehensive programs 
that provide proactive academic, financial, and personal support to students for multiple years 
are an effective way to increase completion rates. Many localities and institutions have begun 
providing forms of financial support such as tuition waivers and emergency aid, and many 
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colleges have begun providing services such as required advising for incoming students and 
structured pathways (which create a route to graduation by outlining courses students must 
take). While these initiatives have merit, each may only moderately improve student outcomes 
on its own. Colleges that seek to dramatically improve graduation rates for low-income students 
can combine them into a package of services with active monitoring and outreach for the dura- 
tion of students’ college careers. 


Integrating services and financial support in this way may be effective for various rea- 
sons. First, many low-income students face multiple barriers to graduation that change over time, 
so providing services consistently over three years helps them over different hurdles. Second, 
messages and outreach from the program make sure students take advantage of services when 
they need them. The ASAP model has been described as providing both an opportunity and an 
obligation for students, meaning that students are provided with a many forms of support, but are 
also informed that it is their responsibility to meet program requirements and to take advantage 
of services. Meetings are supposed to occur regularly and staff members reach out to students if 
students do not show up, which allows those staff members to identify problems before they lead 
students to drop out. Finally, the advising approach improves how students connect with the col- 
lege and its existing services. Program advisers provide students with designated staff members 
who can help them navigate campus and college life. Many students said that although the finan- 
cial support helped them cover the cost of college, advising was the most important aspect of the 
program for them. 


e The ASAP model can be effective in varying localities and institutional 
contexts and for different student populations. 


The Ohio Programs differed from CUNY ASAP in notable ways. First, the programs 
were implemented in Ohio instead of New York, and this geographic difference meant that the 
program had to be modified slightly, as described below. Second, the Ohio colleges are not part 
of a system like CUNY, so the three programs were managed by the institutions as opposed to a 
central office. Finally, the Ohio Programs evaluation included students who both did and did not 
have developmental education requirements and included many nontraditional students, while the 
CUNY ASAP evaluation focused on developmental education students, most of whom were con- 
sidered traditional college students. The fact that the Ohio Programs achieved similar results to 
CUNY ASAP, despite these differences, demonstrates that the ASAP model can be effective in 
varying localities and institutional contexts and for different student populations. This finding is 
noteworthy since practitioners and policymakers have recognized ASAP as a model worth ex- 
panding nationally. 


e Colleges can adjust components of the ASAP model to reflect their local 
contexts and still achieve similarly positive results. 


Because the Ohio colleges were in a different context from CUNY (as described above), 
they put some program components into operation differently. First, students in the Ohio pro- 
grams who met program requirements received monthly $50 gift cards intended for gas or gro- 
ceries while CUNY students received unlimited public transportation cards. Even though the gift 
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cards had more limited monetary value than the unlimited transportation cards, and even though 
the gift cards received mixed reviews from students and staff members, Ohio program group 
students still participated in services at much higher rates than students in the control group, sug- 
gesting the incentive may have met its goal of encouraging participation. It may have also reduced 
nontuition costs such as transportation for students. 


Second, since the Ohio colleges were unable to formally implement blocked course 
schedules as CUNY did, they tried to foster connections among students and facilitate student 
engagement by putting students in classes together or informing students when there were other 
students from the program in their courses. 


Finally, the Ohio Programs also provided tutoring and career services differently: Rather 
than providing those services themselves, the programs made use of existing tutoring and career 
centers on campus and allowed students to fulfill requirements in multiple ways, for example by 
receiving tutoring online. These changes suggest that the program components can be put into 
operation in different ways and still lead to positive results. Since informal, online tutoring may 
be difficult to track, colleges could also partner with their learning centers to identify other ways 
to provide academic support to students. It may matter, however, precisely how the components 
are changed. It is unknown how different changes would affect student outcomes. Since there is 
growing evidence that the ASAP model is effective as designed, colleges should perhaps strive 
to maintain fidelity to the model and only make small tweaks where necessary. 


The Future of ASAP in Ohio and Across the Nation 


One college in Ohio is sustaining its program in its entirety and eventually plans to expand the 
program to serve most if not all of its low-income student population.' Although the demonstra- 
tion period was ending, the college enrolled about 200 new students into the program in the fall 
2018 semester and again in fall 2019. The other two Ohio colleges are not sustaining their pro- 
grams in their entirety but are taking lessons from the programs and applying them to general 
college operations. 


One of the most important factors that affected sustainability decisions was cost. The two 
colleges that decided not to continue their programs cited their high costs. The college that is 
sustaining the program addressed its cost in the short term by repurposing institutional funding 
that aligned with the goals of the program. It continues to explore other long-term funding oppor- 
tunities. Although the Ohio Department of Higher Education (ODHE) engaged the Ohio General 
Assembly, among other entities, in discussions about state-level funding to support the develop- 
ment of comprehensive programs modeled on ASAP in Ohio, the resulting proposal was cut from 
the final state budget.” Later, as this report was being written in 2019, the Ohio General Assembly 
included a budget line that would allow ODHE, with assistance from the Ohio Department of 
Jobs and Family Services, to repurpose funding to establish a Community College Acceleration 


'For more information, see Headlam (2018). 
>Headlam (2018). 
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Program; this program is supposed to provide comprehensive support like that provided by the 
Ohio Programs for students who need it.? 


The fact that only one of the three colleges is continuing to operate its program despite 
the large, positive effects on graduation rates suggests that financial sustainability is an important 
part of the long-term viability of comprehensive student success programs. To develop sustaina- 
ble student success programs, institutions should develop long-term funding plans during early 
program implementation and higher education policymakers should consider public funding 
streams. Notably, the U.S. House of Representatives’ 2019 Higher Education Act reauthorization 
bill (called the College Affordability Act) includes a provision that would provide funding to 
community colleges to develop and implement programs modeled on ASAP.* 


Outside of Ohio and despite funding challenges, ASAP continues to grow both within 
and outside of the CUNY system. With direct funding from the City of New York, CUNY is 
continuing to expand its program and expects to serve 25,000 students across nine colleges in its 
system. Other institutions, such as Westchester Community College in New York, Nashville State 
Community College in Tennessee, and Skyline College in California have begun implementing 
similar programs. MDRC is evaluating the Westchester program and Skyline College is now 
leading the expansion of its program in San Mateo County Community College District.> Study- 
ing these and other programs based on ASAP will continue to build evidence about the imple- 
mentation and effectiveness of comprehensive programs in different geographic areas and for 
different populations of students, and shed light on how institutions and states can improve com- 
munity college completion rates and student success. 


3Ohio 133rd General Assembly, HB 166, 2019. 
4U.S. Congress Education and Labor Committee (2019). 
5San Mateo County Community College District (2019). 
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Appendix A 


Survey Response Bias Analysis and 
the Creation of Survey Scales 


This appendix discusses two aspects of the student survey conducted as part of MDRC’s evalua- 
tion of the Ohio Programs: 


1. Survey response bias analysis, a discussion of the response rate and the potential 
for bias in the results. 


2. The creation of survey scales, a description of two scales created from the student 
survey and reported in Chapter 4 and Appendix D (the quality-of-advising scale and 
the integration-and-sense-of-belonging scale). 


Survey Response Bias Analysis 


This section of the appendix discusses the response rate for the student survey conducted as part 
of MDRC’s evaluation of the Ohio Programs, and the potential for bias in the results. 


Survey Fielding and Respondent Sample 


The student survey asked study participants about a variety of topics including their par- 
ticipation in and experience with student services, their educational experiences, their work ex- 
periences, and their financial situations. The survey was fielded to the 1,431 sample members in 
the fall 2015 and spring 2016 cohorts approximately one year after random assignment. Students 
in the fall 2015 cohort were surveyed September through December 2016. Students in the spring 
2016 cohort were surveyed February through May 2017. A total of 979 responses were collected, 
equivalent to an overall survey response rate of 68 percent. The response rate was very similar 
across the research groups, with 70 percent of program group students and 67 percent of control 
group students responding. 


Characteristics of Survey Respondents 


Three analyses were conducted to test for potential bias in survey responses. First, the 
characteristics of survey respondents were compared with the characteristics of students who did 
not respond to the survey. This comparison provides an indication of how well the survey re- 
spondents represent the full study sample — a form of external validity. Second, academic effects 
through three years were compared for survey respondents, nonrespondents, and the full study 
sample. These comparisons also serve as an indication of external validity. Finally, the character- 
istics of program group students who responded to the survey were compared with characteristics 
of control group students who responded to the survey. This comparison provides an indication 
of whether the results are internally valid for survey respondents. 


Comparison of Respondent and Nonrespondent Baseline Characteristics 


Appendix Table A.1 compares the baseline characteristics of survey respondents and 
nonrespondents. It shows that respondents and nonrespondents differed with respect to gender 
and age: On average, survey respondents were older and a higher percentage were female. 
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Appendix Table A.1 


Characteristics of Survey Respondents and Nonrespondents 


Number of Full Survey Survey 

Characteristic (%) Observations Sample Respondents Nonrespondents P-Value 
Female 1,409 63.5 65.9 58.4 *** — 0.0090 
Has developmental education 
requirements 1,361 73.5 73.2 74.3 0.6470 
Nontraditional 1,417 46.9 46.4 47.9 0.5890 
Had a high school diploma or higher 
at study entry* 1,384 87.4 87.7 86.8 0.6210 
Race/ethnicity 0.5160 

Black 1,385 35.5 35.8 34.9 

Hispanic 1,385 9.5 9.2 10.1 

White 1,385 45.1 45.8 43.6 

Other 1,385 9.9 9.2 11.5 
Age “s 0.0460 

19 or younger 1,420 47.8 46.7 50.3 

20 to 23 1,420 21.5 20.6 23.4 

24 or older 1,420 30.7 32.7 26.3 
Sample size 1,409 965 444 


SOURCES: MDRC calculations using baseline information form data and placement test data from the 
colleges. 


NOTES: A two-tailed t-test was applied to differences between survey response groups for gender, 
developmental requirements, nontraditional student status, and diplomas/degrees earned. A chi-square test 
was applied to differences between the groups for race/ethnicity and age. Levels for statistically significant 
differences between program and control groups are indicated as: * = 10 percent; ** = 5 percent; and *** = 1 
percent. 

To analyze whether on average survey respondents and nonrespondents differed from each other, an 
omnibus F-test was performed, which yielded a p-value less than 0.0001. 

Estimates are adjusted by site and cohort. 

Missing values are not included in individual variable distributions. 

Distributions may not add to 100 percent because of rounding. 

aThis number includes high school diplomas, occupational and technical certificates, and unspecified other 
types of degrees. Not included are students who earned no degree, who earned a high school equivalency and 
no other degrees, or for whom degree information is missing. 


An omnibus F-test was conducted to see whether students’ baseline characteristics were 
jointly predictive of responding to the survey.! The F-test yielded a p-value of less than 0.0001, 
suggesting that respondents and nonrespondents differ significantly in their baseline 


‘Logistic regression was used for this analysis, where the outcome was whether a sample member responded 
to the survey and the predictor variables were students’ baseline characteristics. 
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characteristics. This result suggests that the survey results may not represent the responses non- 
respondents would have given. 


Comparison of Effects on Academic Outcomes for Respondents, 
Nonrespondents, and the Full Sample 


Appendix Table A.2 compares the estimated effects of the Ohio Programs, measured 
using administrative data, for survey respondents and for nonrespondents. For both academic 
outcomes — total credits earned and degrees earned — the table shows that the Ohio Program 
had larger effects for survey respondents than for nonrespondents. Among survey respondents, 
25 percent of control group members earned degrees within three years compared with 42 percent 
of program group members, an estimated effect of 17 percentage points. Among nonrespondents, 
much lower percentages of students earned degrees in both the program and control groups, and 
the estimated effect on earning a degree is 11 percentage points. However, as the final column of 
the table indicates, this difference in estimated effects is not statistically significant. The credit- 
accumulation measure shows a similar pattern: the Ohio Programs’ estimated effect is larger 
among survey respondents than among nonrespondents, and both the program and control groups 
earned fewer credits overall among nonrespondents than among respondents. 


Appendix Table A.2 


Academic Outcomes of Survey Respondents and 
Nonrespondents After Three Years 


P-Value for 
P-Value Differential 
Sample Program Control for Estimated 
Outcome Size Group Group Difference Difference Effects 
Total credits earned 
Full sample 1,431 34.2 26.4 7.8 *** 0.0000 0.6147 
Respondents 979 40.2 32.0 8.2 *** 0.0000 
Nonrespondents 452 21.3 14.4 6.9 *** 0.0005 
Earned a degree from 
any college 
Full sample 1,431 34.3 19.0 15.3 *** 0.0000 0.1329 
Respondents 979 42.4 25.1 17.4 *** 0.0000 
Nonrespondents 452 16.7 5.7 11.0 *** 0.0003 
Sample size 1,431 


SOURCE: MDRC calculations using data from the National Student Clearinghouse and the study colleges. 


NOTES: A two-tailed t-test was applied to differences between research groups. Statistical significance 
levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

A two-tailed t-test was applied to differences in effects between survey respondents and nonrespondents. 
Statistical significance levels are indicated as: ttt = 1 percent; Tt = 5 percent; tT = 10 percent. 

Rounding may cause slight discrepancies in sums and differences. 

Estimates are adjusted by site and cohort. 
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Although there are differences in the magnitude of effects for respondents versus nonre- 
spondents, the differences are less when comparing respondents with the full study sample 
(shown in the top row of each panel). For example, among the full study sample, the estimated 
effect on degree receipt is 15 percentage points — similar in magnitude and statistical significance 
to the 17 percentage point estimated effect among survey respondents. The similar pattern of 
effects suggests that effects for the survey sample are generally representative of effects for the 
full study sample. 


Comparison of Program and Control Group Respondent 
Baseline Characteristics 


As noted above, a slightly higher proportion of program group students responded to the 
survey (70 percent) than control group students (67 percent). Appendix Table A.3 compares the 
baseline characteristics of respondents in the program and control groups to determine whether 


Appendix Table A.3 


Characteristics of Program and Control 
Group Survey Respondents 


Number of All Survey Program Control 

Characteristic (%) Observations Respondents Group Group P-Value 
Female 965 65.6 64.1 67.4 0.2770 
Has developmental education 
requirements 938 72.8 72.4 73.3 0.7420 
Nontraditional 969 46.7 45.0 48.8 0.2260 
Had a high school diploma or higher 
at study entry® 949 87.6 88.4 86.6 0.4280 
Race/ethnicity 0.7010 

Black 949 35.8 35.4 36.3 

Hispanic 949 9.2 8.3 10.2 

White 949 45.8 46.6 44.9 

Other 949 9.2 9.7 8.6 
Age 0.5200 

19 or younger 971 46.7 47.9 45.2 

20 to 23 971 20.6 20.9 20.2 

24 or older 971 32.7 31.2 34.6 
Response rate 1,431 68.4 69.7 66.9 0.2450 
Sample size 965 524 441 


(continued) 
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Appendix Table A.3 (continued) 


SOURCE: MDRC calculations using baseline information form data and placement test data from the 
study colleges. 


NOTES: A two-tailed t-test was applied to differences between survey response groups for gender, 
developmental requirements, nontraditional student status, and diplomas/degrees earned. A chi-square 
test was applied to differences between the groups for race/ethnicity and age. Levels for statistically 
significant differences between program and control groups are indicated as: * = 10 percent; ** = 5 
percent; and *** = 1 percent. 

To analyze whether on average program and control group survey respondents differed from each 
other, an omnibus F-test was performed, which yielded a p-value of 0.1755. This finding suggests that 
relative to the baseline characteristics shown above, program and control group survey respondents do 
not differ from one another. 

Estimates are adjusted by site and cohort. 

Missing values are not included in individual variable distributions. 

Distributions may not add to 100 percent because of rounding. 

aThis number includes high school diplomas, occupational and technical certificates, and unspecified 
other types of degrees. Not included are students who earned no degree, who earned a high school 
equivalency and no other degrees, or for whom degree information is missing. 


respondents’ characteristics differed between the two research groups. The table shows that the 
two groups were comparable, with survey respondents in the program and control groups similar 
with respect to all measured baseline characteristics. 


An omnibus F-test was conducted to see whether survey respondents’ baseline charac- 
teristics were jointly predictive of student’s experimental status. The results were not statistically 
significant, indicating little evidence that the groups of respondents were systematically different 
at the outset of the study. This test confirms that among the survey respondents, it is valid to 
compare the program and control groups. 


Conclusion 


The response rate for the student survey is 68 percent. The characteristics of program and 
control group students who responded to the survey were not significantly different. However, as 
often occurs with surveys, the characteristics of survey respondents were significantly different 
from those of nonrespondents. Nonetheless, the estimated effects among survey respondents are 
very similar to the estimated effects among the full sample, suggesting that the survey findings 
are generally representative of the full sample. 


The Creation of Survey Scales 


This section discusses two scales reported in Chapter 4 and Appendix D, both derived from the 
student survey conducted as part of MDRC’s evaluation of the Ohio Programs: the quality-of- 
advising scale and the integration-and-sense-of-belonging scale. The section covers the questions 
that comprised the scales and the data processing conducted to calculate the values presented in 
Chapter 4 and Appendix D. 
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Quality of Advising 


The quality-of-advising measure presented in Appendix Table D.2 is derived from five 
questions administered in the Ohio Programs one-year student survey. Students were asked to 
indicate if they strongly agreed (1), agreed (2), disagreed (3), or strongly disagreed (4) with the 
following: 


1. You are satisfied in general with the academic advising/coaching you have 
received. 


2. You have received accurate information about courses, programs, and require- 
ments through academic advising. 


3. Academic advisors/coaches kept you informed about deadlines related to institu- 
tional policies and procedures, such as drop/add periods, withdrawal deadlines, 
registration periods, etc. 


4. Academic advising/coaching has been available when you needed it. 
5. Sufficient time has been available when you met with academic advisors/coaches. 


Originally, for all questions, a higher-value response indicated lower-quality advising. 
All five questions were recoded to “change the direction” of responses.” After the questions were 
recoded, higher values indicated higher-quality advising. 


The quality-of-advising scale is the unweighted average of a student’s responses to all 
five questions. Respondents had the option to skip or refuse any of the five questions; a small 
number of students responded to some, but not all five. If the student answered one or two ques- 
tions in the scale, the scale was not calculated. For students who answered three or four questions 
in the scale, the scale was calculated as the average of that student’s responses to the questions 
that were answered. 


Appendix Table D.2 presents the average quality-of-advising scale values among pro- 
gram and control group respondents who reported they had met with an academic adviser during 
the relevant period. 


Integration and Sense of Belonging at School 


The sense-of-belonging measures presented in Figure 4.7 are derived from nine questions 
administered in the Ohio Programs student survey. Students were asked to indicate if they 
strongly agreed (1); agreed (2); disagreed (3); or strongly disagreed (4) with the following: 


1. College is an unfriendly place. 
2. Ido not feel that I fit in or belong in college. 


3. The instructors and staff understand who I am and where I am coming from. 


*For all five questions: 1 became 4, 2 became 3, 3 became 2, 4 became 1. 
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4. It is difficult to make good friends with other students. 
5. The other students do not understand who I am and where I am coming from. 


6. College has the feeling of a community, where many people share the same goals 
and interests. 


7. Many people at college know me by name. 
8. Ido not feel I am part of college life. 


9. I feel that I matter to the college instructors, staff, and other students. 


For some questions, a higher-value response indicates a greater sense of belonging, and 
for others, a higher-value response indicates less of a sense of belonging. Those latter questions 
were recoded to change the “direction” of responses.* After those questions were recoded, higher 
values indicated a greater sense of belonging for all questions. 


The sense-of-belonging scale is the unweighted average of a student’s responses to all 
nine questions. Respondents had the option to skip or refuse any of the nine questions; a small 
number of students responded to some, but not all nine. The plan was to exclude students who 
answered five or fewer of the nine questions in the scale, but all students answered six or more. 


Figure 4.7 presents the percentages of students reporting “high” and “low” senses of be- 
longing. Students in the “low” category are those whose calculated scores are one or more stand- 
ard deviations below the mean, indicating less integration and sense of belonging. Students in the 
“high” category are those whose calculated scores are one or more standard deviations above the 
mean, indicating greater integration and sense of belonging. 


>For questions 3, 6, 7, and 9: 1 became 4, 2 became 3, 3 became 2, 4 became 1. 
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Appendix B 


Baseline Characteristics of Sample Members 


Appendix Table B.1 


Baseline Characteristics of Sample Members, by Research Group 


Characteristic 
Nontraditional® (%) 
Has developmental education requirements (%) 


Intention to enroll (%) 
Full time 
Part time 


Gender (%) 
Male 
Female 


Age (%) 
19 or younger 
20 to 23 
24 or older 


Average age (years) 


Marital status (%) 
Married and living with spouse 
Married and living apart from spouse 
Unmarried and living with partner 
Unmarried and not living with partner 


Living with parents (%) 
Parents pay more than half of expenses (%) 
Missing 


Race/ethnicity” (%) 
Hispanic 
White 
Black 
Other® 
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Full 
Sample 


46.9 


74.3 


90.7 
9.3 


36.1 
63.9 


47.3 
21.7 
30.9 


23.2 


6.8 
1.8 
15.2 
76.3 


57.8 


27.2 
7.5 


9.6 
45.8 
34.8 

9.8 


Program 
Group 


46.1 


74.1 


91.1 
8.9 


37.7 
62.3 


46.9 
22.4 
30.7 


23.0 


7.1 
2.4 
14.1 
76.5 


58.7 


29.0 
7.6 


8.8 
46.9 
35.5 

8.8 


Control 
Group 


47.9 


75.1 


90.4 
9.6 


34.0 
66.0 


47.8 
21.4 
30.8 


23.3 


6.6 
1.1 * 
16.3 
76.0 


56.8 


25.0 * 
7.2 


10.6 
44.9 
34.0 
10.5 


(continued) 


Appendix Table B.1 (continued) 


Full 
Characteristic Sample 
Number of children (%) 
0 73.0 
1 11.7 
2 7.6 
3 or more 7.8 
Mode of transportation to campus (%) 
Driving 70.7 
Carpool 1.9 
Public transportation 15.0 
Drop-off from family member or friend 10.7 
Biking or walking 1.8 
Currently employed (%) 59.9 
Among those currently employed, hours worked per week (%) 
1 - 34 74.0 
35 or more 26.0 
Highest grade completed (%) 
10th or lower 46 
11th 4.9 
12th® 90.6 
Diplomas/degrees earned® (%) 
High school diploma 87.2 
High school equivalency 12.1 
Occupational/technical certificate 11.3 
Other 1.9 
Date of high school graduation/equivalency receipt (%) 
Within the past two years 57.9 
More than two years ago 42.1 
Highest degree student plans to attain (%) 
Associate's 19.4 
Bachelor's 41.0 
Master's 26.4 
Professional or doctorate 13.2 
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Program 
Group 


73.6 
10.9 
8.1 
7.5 


72.3 
1.9 
14.5 
9.5 
1.8 


57.6 


74.2 
25.8 


4.2 
5.5 
90.3 


87.4 
12.3 
9.7 
2.3 


57.4 
42.6 


19.4 
42.3 
25.7 
12.6 


Control 
Group 


72.1 
12.5 
7.5 
7.9 


68.5 
1.8 
16.0 
11.9 
1.7 


61.7 


74.0 
26.0 


5.0 
4.0 
90.9 


86.9 
11.9 
13.0 * 
1.4 


58.6 
41.4 


19.5 
39.9 
27.2 
13.4 


(continued) 


Appendix Table B.1 (continued) 


Full Program Control 
Characteristic Sample Group Group 
First person in the family to attend college (%) 33.9 34.8 33.0 
Highest degree/diploma earned by the student's mother (%) 
Not a high school graduate 11.9 12.8 11.1 
High school diploma or equivalency 34.1 33.2 35.3 
Some college, did not complete a degree 19.8 20.7 18.7 
College degree (AA, BA, MA, PhD) 25.6 24.5 26.5 
Missing 8.6 8.8 8.4 
Highest degree/diploma earned by the student's father (%) 
Not a high school graduate 15.8 15.8 15.9 
High school diploma or equivalency 38.7 39.2 38.0 
Some college, did not complete a degree 12.7 12.4 13.0 
College degree (AA, BA, MA, PhD) 13.5 14.1 12.8 
Missing 19.3 18.5 20.3 
Language other than English spoken regularly at home (%) 8.6 8.5 8.9 
Sample size 1,501 806 695 


SOURCE: MDRC calculations using baseline information form data and placement test data from the 
study colleges. 


NOTES: Italics indicate statistics calculated only for a subset of respondents. 

Distributions may not add to 100 percent because of rounding. 

Missing values are included in variable distributions only for characteristics with more than 6 percent of 
the full sample missing. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels 
are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

To analyze whether program and control group survey respondents differed from each other on 
average, an omnibus F-test was performed, which yielded a p-value of 0.533. This finding suggests that 
relative to the baseline characteristics shown above, program and control group survey respondents do 
not differ from one another. 

aNontraditional students are defined as those who were 24 or older, worked 35 or more hours per 
week, had children, or did not receive a high school diploma and were not enrolled in high school at the 
time of random assignment. Students are listed as nontraditional if they fit any of these characteristics. 
Students are considered to be missing data in the nontraditional category if they were missing data on two 
or more of these variables and have no other nontraditional characteristic; however, since less than 6 
percent of the study sample were missing data, this percentage is not listed in the table. 

bRespondents who said they were Hispanic and chose a race are included only in the "Hispanic" 
category. 

°The "other" category includes Asian/Pacific Islander, Native American/Alaska Native, multiracial, and 
other races and ethnicities. 

4This category includes students who were enrolled in high school at the time of random assignment. 

Distributions may not add to 100 percent because categories are not mutually exclusive. 
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Appendix Table B.2 


Baseline Characteristics of Sample Members, by College 


Characteristic 


Program status (%) 
Program group 
Control group 


Nontraditional? (%) 
Has developmental education requirements (%) 


Intention to enroll (%) 
Full time 
Part time 


Gender (%) 
Male 
Female 


Age (%) 
19 or younger 
20 to 23 
24 or older 


Average age (years) 


Marital status (%) 
Married and living with spouse 
Married and living apart from spouse 
Unmarried and living with partner 
Unmarried and not living with partner 


Living with parents (%) 
Parents pay more than half of expenses (%) 
Missing 


Race/ethnicity” (%) 
Hispanic 
White 
Black 
Other® 


86 


Full 
Sample 


53.7 
46.3 


46.9 


74.3 


90.7 
9.3 


36.1 
63.9 


47.3 
21.7 
30.9 


23.2 


6.8 
1.8 
15.2 
76.3 


57.8 


27.2 
7.5 


9.6 
45.8 
34.8 

9.8 


Cincinnati 
State 


54.6 
45.4 


58.7 


61.0 


87.9 
12.1 


37.4 
62.6 


29.3 
26.5 
44.2 


24.8 


7.2 
2.8 
18.7 
71.4 


42.3 


16.1 
7.3 


3.1 
34.4 
51.3 
11.2 


Lorain 


56.9 
43.1 


39.4 


81.2 


91.6 
8.4 


34.1 
65.9 


57.5 
17.3 
25.1 


22.2 


7.3 
1.2 
14.2 
77.3 


63.1 


31.2 
5.5 


16.7 
55.5 
19.0 

8.8 


Tri-C 


49.7 
50.3 


43.8 


79.6 


92.3 
7.7 


37.0 
63.0 


53.4 
21.8 
24.8 


22.6 


5.9 
1.4 
13.1 
79.6 


66.4 


33.4 
9.6 


8.0 
46.3 
36.2 

9.5 


(continued) 


Appendix Table B.2 (continued) 


Full Cincinnati 
Characteristic Sample State Lorain Tri-C 
Number of children (%) 
0 73.0 64.8 73.8 79.6 
1 11.7 16.0 11.6 7.8 
2 7.6 9.0 7.5 6.4 
3 or more 7.8 10.3 7.1 6.2 
Mode of transportation to campus (%) 
Driving 70.7 64.6 80.2 66.6 
Carpool 1.9 1.5 2.5 1.6 
Public transportation 15.0 24.0 1.2 20.8 
Drop-off from family member or friend 10.7 8.2 14.5 9.1 
Biking or walking 1.8 1.7 1.6 2.0 
Currently employed (%) 59.9 62.1 59.3 58.5 
Among those currently employed, hours worked per week (%) 
1 - 34 74.0 72.1 81.8 67.8 
35 or more 26.0 27.9 18.2 32.2 
Highest grade completed (%) 
10th or lower 4.6 5.1 4.3 4.4 
11th 4.9 5.5 4.2 5.0 
12th® 90.6 89.4 91.5 90.6 
Diplomas/degrees earned® (%) 
High school diploma 87.2 84.9 89.2 87.3 
High school equivalency 12.1 14.4 10.6 11.5 
Occupational/technical certificate 11.3 10.7 10.8 12.3 
Other 1.9 2.2 1.4 2.2 
Date of high school graduation/equivalency receipt (%) 
Within the past two years 57.9 39.6 67.4 64.7 
More than two years ago 42.1 60.4 32.6 35.3 
Highest degree student plans to attain (%) 
Associate's 19.4 14.4 23.3 20.0 
Bachelor's 41.0 41.8 42.2 39.1 
Master's 26.4 27.9 25.7 25.9 
Professional or doctorate 13.2 16.0 8.8 15.0 


(continued) 
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Appendix Table B.2 (continued) 


Full Cincinnati 
Characteristic Sample State Lorain Tri-C 
First person in the family to attend college (%) 33.9 36.5 30.8 34.7 
Highest degree/diploma earned by the student's mother (%) 
Not a high school graduate 11.9 14.6 9.7 11.7 
High school diploma or equivalency 34.1 32.3 37.4 32.4 
Some college, did not complete a degree 19.8 16.9 21.1 21.1 
College degree (AA, BA, MA, PhD) 25.6 27.8 25.5 23.6 
Missing 8.6 8.4 6.2 11.1 
Highest degree/diploma earned by student's father (%) 
Not a high school graduate 15.8 16.7 15.2 15.5 
High school diploma or equivalency 38.7 33.6 44.8 37.2 
Some college, did not complete a degree 12.7 13.9 13.5 10.7 
College degree (AA, BA, MA, PhD) 13.5 16.7 11.1 13.1 
Missing 19.3 19.1 15.4 23.4 
Language other than English spoken regularly at home (%) 8.6 10.2 6.5 9.3 
Sample size 1,501 467 513 521 


SOURCE: MDRC calculations using baseline information form data and placement test data from the study 
colleges. 


NOTES: Cincinnati State = Cincinnati State Technical and Community College; Lorain = Lorain County Community 
College; Tri-C = Cuyahoga Community College. 

Italics indicate statistics calculated only for a subset of respondents. 

Distributions may not add to 100 percent because of rounding. 

Missing values are included in variable distributions only for characteristics with more than 6 percent of the full 
sample missing. 

aNontraditional students are defined as those who were 24 or older, worked 35 or more hours per week, had 
children, or did not receive a high school diploma and were not enrolled in high school at the time of random 
assignment. Students are listed as nontraditional if they fit any of these characteristics. Students are considered to 
be missing data in the nontraditional category if they were missing data on two or more of these variables and have 
no other nontraditional characteristic; however, since less than 6 percent of the study sample were missing data, 
this percentage is not listed in the table. 

>Respondents who said they were Hispanic and chose a race are included only in the "Hispanic" category. 

¢The "other" category includes Asian/Pacific Islander, Native American/Alaska Native, multiracial, and other 
races and ethnicities. 

dThis category includes students who were enrolled in high school at the time of random assignment. 

Distributions may not add to 100 percent because categories are not mutually exclusive. 
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Appendix C 


Program Participation Rates 


Appendix Table C.1 


Participation in Program Activities 


Outcome 
Enrolled in classes in the first semester 


Among those enrolled: 
Met with an adviser 


Academic advising appointments attended 
0 
1to5 
6 or more 


Met with a career adviser 
Received a financial incentive 


Financial incentives received 
0 
1 
2 
3 or more 


Enrolled in classes in the second semester 


Among those enrolled: 
Met with an adviser 


Academic advising appointments attended 
0 
1to5 
6 or more 


Met with a career adviser 


Received a financial incentive 


Financial incentives received 
0 
1 
2 
3 or more 


Enrolled in classes in the third semester 


Among those enrolled: 
Met with an adviser 
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Percentage of the Program 
Group 


95.0 


95.2 


4.8 
34.3 
60.8 


45.1 


81.2 


18.8 
13.8 
18.9 
48.5 


78.6 


93.7 


6.3 
49.0 
44.7 


61.8 


76.1 


23.9 
10.4 
19.9 
45.8 


67.7 


93.7 


(continued) 


Appendix Table C.1 (continued) 


Outcome 


Academic advising appointments attended 
0 
1to5 
6 or more 


Met with a career adviser 


Received a financial incentive 


Financial incentives received 
0 
1 
2 
3 or more 


Enrolled in classes in the fourth semester 


Among those enrolled: 
Met with an adviser 


Academic advising appointments attended 
0 
1to5 
6 or more 


Met with a career adviser 


Received a financial incentive 


Financial incentives received 
0 
1 
2 
3 or more 


Enrolled in classes in the fifth semester 


Among those enrolled: 
Met with an adviser 


Academic advising appointments attended 
0 
1to5 
6 or more 
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Percentage of the Program 
Group 


6.3 
57.9 
35.8 


68.7 


77.7 


22.3 
16.1 
16.4 
45.2 


58.6 


89.4 


10.6 
56.0 
33.4 


65.8 


74.9 


25.1 

9.5 
12.3 
53.2 


48.7 


84.1 


15.9 

60.5 

23.7 
(continued) 


Appendix Table C.1 (continued) 


Percentage of the Program 


Outcome Group 
Met with a career adviser 64.7 
Received a financial incentive 71.6 


Financial incentives received 


0 28.4 
1 8.9 
2 9.7 
3 or more 53.0 
Enrolled in classes in the sixth semester 41.5 


Among those enrolled: 
Met with an adviser 73.7 


Academic advising appointments attended 


0 21.3 
1to5 56.2 
6 or more 22.5 
Met with a career adviser 58.3 
Received a financial incentive 65.4 


Financial incentives received 


0 34.6 
1 9.3 
2 12.4 
3 or more 43.7 
Sample size 806 


SOURCE: MDRC calculations using data from the MDRC management information system. 
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Appendix Table C.2 


Participation in Tutoring 


Outcome 


Enrolled in developmental education in the first semester 


Among those enrolled in developmental education: 
Attended a tutoring session (at least once) 


Hours of tutoring attended 
0 
Less than 3 
3-8.9 
9 or more 


Enrolled in developmental education in the second semester 


Among those enrolled in developmental education: 
Attended a tutoring session (at least once) 


Hours of tutoring attended 
0 
Less than 3 
3 - 8.9 
9 or more 


Enrolled in developmental education in the third semester 


Among those enrolled in developmental education: 
Attended a tutoring session (at least once) 


Hours of tutoring attended 
0 
Less than 3 
3 - 8.9 
9 or more 


Enrolled in developmental education in the fourth semester 


Among those enrolled in developmental education: 
Attended a tutoring session (at least once) 


Hours of tutoring attended 
0 
Less than 3 
3 - 8.9 
9 or more 
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Percentage of the Program 


Group 


52.8 


58.1 


42.1 

9.0 
17.8 
31.1 


24.2 


71.8 


28.2 
13.7 
22.0 
36.0 


8.0 


56.2 


43.8 
11.5 
25.4 
19.3 


4.1 


37.8 


62.2 
6.5 
21.6 
9.8 


(continued) 


Appendix Table C.2 (continued) 


Outcome 


Enrolled in developmental education in the fifth semester 


Among those enrolled in developmental education: 
Attended a tutoring session (at least once) 


Hours of tutoring attended 
0 
Less than 3 
3 - 8.9 
9 or more 


Enrolled in developmental education in the sixth semester 


Among those enrolled in developmental education: 
Attended a tutoring session (at least once) 


Hours of tutoring attended 
0 
Less than 3 
3 - 8.9 
9 or more 


Sample size 


Percentage of the Program 
Group 


3.3 


43.1 


56.9 

0.0 
30.3 
12.9 


1.3 


43.6 


56.4 
10.9 
21.6 
11.1 


806 


SOURCE: MDRC calculations using data from the MDRC management information system. 
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Appendix Table C.3 


Students’ First-Year Experiences with Program Components 


Percentage of 
the Program 


Outcome Group 
How would you evaluate your overall experience in the program? (%) 
Excellent 67.0 
Good 24.8 
Fair 5.8 
Poor 2.4 
Received a $50 gift card (%) 74.1 


Among those who received $50 gift cards 
Used gift card to purchase (%) 


Food 74.0 
Gas 53.6 
Other 24.1 
Liked the $50 gift card as an incentive for completing the program requirements (%) 90.8 


Among those who did not like the gift card as an incentive 
What did you not like about the gift card? (%) 


The store 22.6 
The amount 20.6 
Something else 74.0 


Among those who did not like the store 
What kind of gift card would you prefer? (%) 


One you could use anywhere, such as Visa, Mastercard, Discover gift card 87.1 

Amazon gift card 38.7 

iTunes gift card 0.0 

A different store 22.5 
Among those who did not like the amount of the gift card 

How much money do you think would be appropriate for completing the requirements? ($) 226.2 

Total program respondents 533 


SOURCE: MDRC calculations using data from the student survey. 


NOTE: Rounding may cause slight discrepancies in sums and differences. 


96 


Appendix D 


Student Survey Findings 


Appendix Table D.1 


Students’ First-Year Experiences: Messages Received 


Sample Program Control Standard 
Response Size Group Group Difference Error 
Often or very often heard college 
faculty/staff members speak about: 
The importance of enrolling in school full time 947 73.8 45.6 28.2*** 3.2 
That it is a good idea to enroll in school during 
the summer semesters 948 66.0 40.2 25.9*** 3.2 
About the goal of obtaining an associate’s degree 
within 3 years 948 72.8 45.0 27.8*** 3.2 
Survey sample size 978 533 445 


SOURCE: MDRC calculations using data from the student survey. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

A two-tailed t-test was applied to differences between research groups. 

Estimates are adjusted by site, cohort, gender, race/ethnicity, age, parental status, marital status, weekly hours 
worked, dependence on parents for 50 percent or more of financial support, whether a student is the first family 
member to attend college, whether a student earned a high school diploma, the number of developmental educa- 
tion requirements at the time of random assignment, and intended enrollment level. 
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Appendix Table D.2 


Students’ First-Year Experiences: Advising 


Sample Program Control Standard 
Response Size Group Group Difference Error 
Ever spoke with an adviser (%) 979 87.7 79.2 8.5*** 2.5 
Average number of times spoke with an adviser 
in the first year 
First semester 920 13.2 4.8 8.4*** 0.8 
Second semester 812 13.9 3.8 10.1*** 1.0 
Among those who spoke with an adviser: 
In what ways have you interacted with an adviser? (%) 
In-person meetings 830 98.4 96.4 
Phone calls 830 49.3 35.8 
Emails 830 64.7 39.4 
Text messages 830 12.5 3.3 
Social media 830 3.9 0.9 
Average time spent during a visit to an adviser (%) 
15 minutes or fewer 828 18.9 29.5 
16-30 minutes 828 53.5 46.0 
31 minutes or more 828 27.6 24.6 
Average number of topics discussed with an adviser 827 6.9 5.3 
Topics discussed with an adviser: 
Academic goals 827 89.2 81.0 
Academic progress 827 86.5 74.8 
Course selection 827 85.9 85.7 
Major 827 75.9 65.2 
Requirements for graduation 827 72.5 60.2 
Internships 827 29.7 18.5 
Job opportunities 827 34.1 17.8 
Career planning 827 48.8 25.4 
Transfer credit policies, probation, 
and add/drop policies 827 52.17 32.8 
College services such as financial aid, 
tutoring, and counseling 827 65.5 45.5 
Personal matters 827 48.0 19.8 
Something else 827 6.7 27 
Quality-of-advising scale (avg.) 827 3.6 3.3 
Survey sample size 978 533 445 


SOURCE: MDRC calculations using data from the student survey. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 
Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 
A two-tailed t-test was applied to differences between research groups. 
Estimates are adjusted by site, cohort, gender, race/ethnicity, age, parental status, marital status, weekly hours 

worked, dependence on parents for 50 percent or more of financial support, whether a student is the first family 

member to attend college, whether a student earned a high school diploma, the number of developmental educa- 
tion requirements at the time of random assignment, and intended enrollment level. 
The quality-of-advising scale was calculated as the average of a student’s responses to five questions about the 


quality of advising received. 
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Appendix Table D.3 


Students’ First-Year Experiences: Tutoring 


Sample Program Control Standard 

Response Size Group Group Difference Error 
Ever received tutoring in class or outside of class 950 63.3 44.5 18.8*** 3.2 
Average number of times used tutoring services 
outside of class 

First semester 927 8.3 3.3 5.0*** 0.8 

Second semester 819 7.6 3.4 4.2" 0.8 
Survey sample size 978 533 445 


SOURCE: MDRC calculations using data from the student survey. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

A two-tailed t-test was applied to differences between research groups. 

Estimates are adjusted by site, cohort, gender, race/ethnicity, age, parental status, marital status, weekly hours 
worked, dependence on parents for 50 percent or more of financial support, whether a student is the first family 
member to attend college, whether a student earned a high school diploma, the number of developmental educa- 
tion requirements at the time of random assignment, and intended enrollment level. 
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Appendix Table D.4 


Students’ First-Year Experiences: Career Services 


Sample Program Control Standard 
Response Size Group Group Difference Error 
Ever met with career or employment services staff (%) 945 61.8 27.7 34.0*** 3.1 
Average number of times spoke with career or 
employment services staff 
First semester 920 2.4 0.6 1.8*** 0.3 
Second semester 816 3.1 0.6 2.5*** 0.6 
Among those who met with career 
or employment services 
What career services or planning activities have 
you participated in? 
Attended one-on-one meetings with 
career services 424 67.2 55.9 
Enrolled in a career-planning course 424 24.5 24.3 
Attended career seminars or group meetings 424 47.8 32.3 
Completed a career-planning activity 424 63.0 52.3 
Survey sample size 978 533 445 


SOURCE: MDRC calculations using data from the student survey. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

A two-tailed t-test was applied to differences between research groups. 

Estimates are adjusted by site, cohort, gender, race/ethnicity, age, parental status, marital status, weekly hours 
worked, dependence on parents for 50 percent or more financial support, whether a student is the first family 
member to attend college, whether a student earned a high school diploma, the number of developmental educa- 
tion requirements at the time of random assignment, and intended enrollment level. 
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Appendix Table D.5 


Participation in Academic Program Components 


Sample Program Control Standard 
Response Size Group Group Difference Error 
Enrolled in a course with at least 4 other students 
in the same research group 1,501 49.0 23.7 25.3*** 2.2 
First semester 1,501 32.3 20.1 12.1*** 2.0 
Second semester 1,501 17.7 4.4 13.3*** 1.5 
Third semester 1,501 11.3 1.0 10.3*** 1.2 
Fourth semester 1,501 5.6 0.0 iy dpe 0.8 
Fifth semester 1,501 3.7 0.0 ro ai 0.7 
Sixth semester 1,501 1.0 0.0 1.0*** 0.4 
Ever enrolled in a first-year experience course 1,501 34.3 33.7 0.6 2.2 
Sample size 1,501 806 695 


SOURCE: MDRC calculations using transcript data from the study colleges. 


NOTES: Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Enrollment in spring and summer semesters are combined. 

A two-tailed t-test was applied to differences between research groups. 

Estimates are adjusted by site, cohort, gender, race/ethnicity, age, parental status, marital status, weekly hours 
worked, dependence on parents for 50 percent or more of financial support, whether a student is the first family 
member to attend college, whether a student earned a high school diploma, the number of developmental educa- 
tion requirements at the time of random assignment, and intended enrollment level. 
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Appendix Table D.6 


Textbook Stipend Receipt 


Program —_ Control Standard 
Outcome Group Group _ Difference Error 
Received all textbooks free of charge 39.2 18.9 20.4 *** 2.8 
College 1 16.7 3.6 13.1:"%" 3.4 
College 2 58.2 24.3 33.9 *** 5.0 
College 3 41.9 26.7 15.1 ** 6.3 
Received all or most textbooks free of charge 59.1 28.4 30.7 *** 3.0 
Sample size 514 426 


SOURCE: MDRC calculations using data from the student survey and financial aid data from the study 
colleges. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by site, cohort, gender, race/ethnicity, age, parental status, marital status, weekly 
hours worked, dependence on parents for 50 percent or more of financial support, whether a student is the 
first family member to attend college, whether a student earned a high school diploma, the number of 
outstanding developmental education requirements at the time of random assignment, and intended 
enrollment level. 
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Appendix Table D.7 


Students’ First-Year Experiences: Finances 


Sample Program Control Standard 
Response Size Group Group Difference Error 
Funding sources for college (%) 
Ohio Programs financial resources 936 75.7 7.1 68.6*** 2.3 
External financial aid 936 93.3 92.8 0.5 1.7 
Student loans 936 32.2 34.9 -2.7 2.8 
Parents, relatives, partners, or friends 936 11.4 19.4 -8.0*** 2.4 
Employment 936 29.5 33.8 -4.3 3.0 
Credit cards 936 7.0 8.4 -1.5 1.8 
Personal savings 936 18.1 25.4 -7,.3°"* 2.7 
Other 936 4.2 1.7 2.5* 1.1 
Chose not to register for college full time because 
could not afford to (%) 943 15.8 31.6 -15.8*** 2.8 
Often or very often did not buy textbooks because 
of the cost (%) 941 14.3 20.4 -6.1** 2.6 
Missed class often or very often because 
needed to work (%) 939 10.4 10.3 0.1 2.0 
Missed class often or very often because could not 
pay for transportation (%) 940 7.8 11.2 -3.4* 2.0 
Had few or no concerns about ability to 
pay for college (%) 939 46.6 32.6 14.0*** 3.1 
Had a job (%) 
First semester (%) 936 67.7 67.2 0.6 2.7 
Second semester (%) 936 67.1 67.5 -0.4 2.8 
Among those who worked, average hours 
worked per week 
First semester 625 29.8 31.3 
Second semester 627 30.0 31.4 
Survey sample size 978 533 445 


SOURCE: MDRC calculations using data from the student survey. 


NOTES: Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Enrollment in spring and summer semesters are combined. 

A two-tailed t-test was applied to differences between research groups. 

Estimates are adjusted by site, cohort, gender, race/ethnicity, age, parental status, marital status, weekly hours 
worked, dependence on parents for 50 percent or more of financial support, whether a student is the first family 
member to attend college, whether a student earned a high school diploma, the number of developmental educa- 
tion requirements at the time of random assignment, and intended enrollment level. 
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Appendix Table D.8 


Students’ First-Year Experiences: Engagement in College 


Sample Program Control Standard 

Response Size Group Group Difference Error 
Integration and sense of belonging at school 

Low 947 14.3 19.7 -5.4** 2.5 

High 947 14.9 14.4 0.6 2.4 
Had a college employee to turn to for advice 946 88.9 72.4 16.5*** 2.6 
Had all or most services and support 
needed to succeed 948 91.1 80.7 10.4*** 2.3 
Highest degree student planned to earn 

No degree 973 4.1 4.9 -0.7 1.4 

Associate’s 973 22.2 23.5 -1.4 2.7 

Bachelor's 973 41.7 37.0 4.7 3.2 

Master's 973 22.0 23.3 -1.3 2.8 

Professional or doctorate 973 10.0 11.4 -1.3 2.0 
Survey sample size 978 533 445 


SOURCE: MDRC calculations using data from the student survey. 


NOTES: Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Enrollment in spring and summer semesters are combined. 

A two-tailed t-test was applied to differences between research groups. 

Estimates are adjusted by site, cohort, gender, race/ethnicity, age, parental status, marital status, weekly hours 
worked, dependence on parents for 50 percent or more of financial support, whether a student is the first family 
member to attend college, whether a student earned a high school diploma, the number of developmental educa- 
tion requirements at the time of random assignment, and intended enrollment level. 

The integration-and-sense-of-belonging scale was calculated as the average of a student’s responses to nine 
questions about sense of integration with and belonging to the school community. Students in the “low” category 
are those whose calculated scores are one or more standard deviations below the mean, indicating less integra- 
tion and sense of belonging; students in the “high” category are those whose calculated scores are one or more 
standard deviations above the mean, indicating a greater integration and sense of belonging. 
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Appendix E 


Subgroup Effects 


Appendix Table E.1 


Total Credits Earned After Three Years: 
Variation in Effects, by Baseline Student Characteristics 


Baseline Characteristic 


Study college 
College 1 
College 2 
College 3 


Sample size 


Developmental education needs 
With developmental education needs 
Without developmental education needs 


Sample size 


Gender 
Female 
Male 


Sample size 


Ethnicity/race 
Black 
Hispanic 
White 
Other 


Sample size 


Age category 
19 or younger 
20 - 23 
24 or older 


Sample size 


High school diploma or equivalency 
High school diploma 
High school equivalency 


Sample size 


Sample 
Size 


521 
467 
513 


1,501 


1,060 
366 


1,426 


945 
534 


1,479 


507 
139 
667 
142 


1,455 


705 
324 
461 


1,490 


1,268 
163 


1,431 


Average Credits Earned 


Program 
Group 


38.4 
28.5 
36.5 


34.3 
38.5 


36.6 
32.5 


27.4 
38.1 
41.3 
30.8 


37.5 
31.9 
32.5 


35.4 
31.2 
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Control 


Group Difference 


30.2 
23.5 
25.9 


25.2 
30.2 


26.1 
27.2 


21.3 
26.6 
31.5 
23.4 


27.6 
24.8 
26.6 


26.5 
25.2 


8.2 kK 
5.1 ** 
10.6 *** 


9.1 KK 
8.3 RK 


10.5 *** 
5.4 ** 


6.1 ** 

11.5 ** 
oo ats 
7.4 * 


99 kK 
TMs 
5.9 ** 


8.9 KK 
6.0 


P-Value 
for 
Difference 


0.0000 
0.0260 
0.0000 


0.0000 
0.0020 


0.0000 
0.0130 


0.0030 
0.0220 
0.0000 
0.0950 


0.0000 
0.0140 
0.0120 


0.0000 
0.1400 


P-Value for 
Differential 
Estimated 
Effects 


0.2350 


0.8090 


0.0540 fT 


0.5380 


0.3790 


0.4950 


(continued) 


Baseline Characteristic 


Traditional vs. nontraditional 
Traditional 
Nontraditional 


Sample size 


Semester of cohort 
Spring cohort 
Fall cohort 


Sample size 


Appendix Table E.1 (continued) 


Sample 
Size 


789 
698 


1,487 


652 
849 


1,501 


Average Credits Earned P-Value 
Program Control for 
Group Group Difference Difference 
36.6 28.1 8.6 *** 0.0000 
33.0 24.8 8.2 *** 0.0000 
34.4 22.7 11.7 *** 0.0000 
35.1 29.4 5.8 *** 0.0010 


P-Value for 
Differential 
Estimated 
Effects 


0.9020 


0.0200 tt 


SOURCE: MDRC calculations using baseline information form data, placement test data, and transcript data from the study 


colleges. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 
A two-tailed t-test was applied to differences between research groups. Statistical significance levels are indicated 
as: *** = 1 percent; ** = 5 percent; * = 10 percent. 
A two-tailed t-test was applied to differences in effects between subgroups. Statistical significance levels are indicated as: 
ttt = 1 percent; tt = 5 percent; + = 10 percent. 
Estimates are adjusted by site, cohort, gender, race/ethnicity, age, parental status, marital status, weekly hours worked, 
dependence on parents for 50 percent or more of financial support, whether a student is the first family member to attend 
college, whether a student earned a high school diploma, the number of developmental education requirements at the time of 
random assignment, and intended enrollment level. 
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Appendix Table E.2 


Earning a Degree at Any College After Three Years: 
Variation in Effects, by Baseline Student Characteristics 


Percentage Who 


Earned Degrees P-Value 
Sample Program Control for 
Baseline Characteristic Size Group Group Difference Difference 
Study college 
College 1 521 36.2 18.2 18.0 *** 0.0000 
College 2 467 34.3 23.4 10.9 *** 0.0090 
College 3 513 33.2 17.6 15.6 *** 0.0000 
Sample size 1,501 
Developmental education needs 
With developmental education needs 1,060 32.0 16.3 15.7 *** 0.0000 
Without developmental education needs 366 45.0 28.0 17.0 *** 0.0010 
Sample size 1,426 
Gender 
Female 945 37.9 19.6 18.3 *** 0.0000 
Male 534 29.6 20.1 9.5 ** 0.0110 
Sample size 1,479 
Ethnicity/race 
Black 507 25.7 14.6 ras ea 0.0020 
Hispanic 139 38.1 18.8 19.3 ** 0.0300 
White 667 44.1 23.9 20.2 *** 0.0000 
Other 142 25.3 15.0 10.3 0.1780 
Sample size 1,455 
Age category 
19 or younger 705 35.1 18.3 16.8 *** 0.0000 
20 - 23 324 31.8 21.3 10.5 *™* 0.0420 
24 or older 461 36.1 20.7 15.4 *** 0.0000 
Sample size 1,490 
High school diploma or equivalency 
High school diploma 1,268 35.3 20.1 15:3°"7 0.0000 
High school equivalency 163 30.7 12.7 18.0 ** 0.0150 


Sample size 1,431 


111 


P-Value for 
Differential 
Estimated 
Effects 


0.4640 


0.8220 


0.0610 fT 


0.2680 


0.5770 


0.7190 


(continued) 


Appendix Table E.2 (continued) 


Percentage Who P-Value for 
Earned Degrees P-Value Differential 
Sample Program Control for Estimated 
Baseline Characteristic Size Group Group Difference Difference Effects 
Traditional vs. nontraditional 0.7020 
Traditional 789 34.2 19.5 14.7 *** 0.0000 
Nontraditional 698 35.7 19.3 16.5 *** 0.0000 
Sample size 1,487 
Semester of cohort 0.7540 
Spring cohort 652 33.8 17.3 16.5 *** 0.0000 
Fall cohort 849 35.8 20.7 15.1 *** 0.0000 
Sample size 1,501 


SOURCE: MDRC calculations using data from the National Student Clearinghouse, baseline information form data, 
placement test data, and transcript data from the study colleges. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are indicated 
as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

A two-tailed t-test was applied to differences of impacts between subgroups. Statistical significance levels are indicated 
as: ttt = 1 percent; tt = 5 percent; t = 10 percent. 

Estimates are adjusted by site, cohort, gender, race/ethnicity, age, parental status, marital status, weekly hours 
worked, dependence on parents for 50 percent or more of financial support, whether a student is the first family member 
to attend college, whether a student earned a high school diploma, the number of developmental education requirements 
at the time of random assignment, and intended enrollment level. 
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Appendix F 


The Ohio Programs’ Costs 


This appendix supplements the information provided in the main body of the report by providing 
additional cost calculations, including the direct cost per enrolled student and the steady-state 
direct costs per program member and per enrolled student. 


Direct Cost Per Enrolled Student Per Year 


The cost per program group member may be of interest to those seeking to create a budget for the 
three-year costs of operating this type of program for an incoming group of students. It is also 
useful for aligning net costs with the effects of the programs. However, the cost per program 
group member obfuscates the amount spent on enrolled students, since it includes all students 
(enrolled and unenrolled) in the calculation, and many students drop out or graduate within two 
years. Consequently, some readers may be interested in understanding the cost per enrolled stu- 
dent per year, since at many colleges a large proportion of revenue is associated with enrollment. 
Moreover, a college seeking to sustain a program of this type may want the cost of serving a 
particular number of students per year, with the plan of filling program slots as students drop out 
or graduate. 


The cost per program group member is lower than the cost per student participating in 
the program because many students assigned to the program do not remain enrolled throughout 
the program period. The cost per participating student per semester enrolled was calculated using 
enrollment data for program group students. The number of semesters that each student took 
courses was averaged to attain this number (about three). This amount serves as a proxy of cost 
per program participant since students who do not enroll are not receiving the program. This 
figure is multiplied by two to give the average annual direct cost per program participant. Using 
this method, the direct cost is $3,339 per enrolled student per year, almost exactly the same as the 
per-enrollee amount calculated in the interim report, or about $10,000 total over three years. The 
amount per year is about $1,500 higher than the direct cost per program group student per year 
(that is, including students who did not enroll).! 


Steady-State Direct Cost 


To estimate the direct program cost per student to be expected once the program is up and running 
(that is, not in its preliminary phase or in a wind-down phase), steady-state costs were calculated 
using data from the program management information system. The steady-state period chosen 
was the second year of the study, the 2015-2016 academic year. The following were estimated 
for each semester of the program: 


e Number of adviser visits 
e Hours of tutoring 


‘Base costs for students who would be enrollees if given the opportunity to be in the program are not esti- 
mated, nor are net costs for enrollees. Credits attempted by the equivalent control group students (those who 
would have enrolled if they had been in the program group) are not directly observable because the intervention 
affected which students enrolled, even in the first semester. 
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e Number of career counseling visits 


For the steady-state period, the amount spent on each of these activities was divided by 
the relevant units of the activity (for example, number of adviser visits). Administrative costs 
from the second year not clearly associated with any of these three activities were divided by the 
sum of all three activities in the second year. The steady-state costs per activity from the second 
year of the study were then multiplied by the total amounts of those activities provided in all 
semesters of the study. The direct costs of forms of financial support such as book vouchers and 
tuition waivers were then added to the result to yield the overall steady-state total cost estimate: 
$1,451 per program group student per year, $4,353 over three years ($2,633 per enrolled student 
per year, $7,899 over three years). These amounts should be closer to the costs expected for pro- 
grams that are up and running. 


Steady-state cost per adviser visit * total adviser visits + 
Steady-state cost per hour of tutoring * total hours of tutoring + 
Steady-state cost of a career counselor visit * total career visits + 
Steady-state administration cost per student * all enrolled program students ct: 


Direct costs of financial support 


Total steady-state cost 
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About MDRC 


MDRC is a nonprofit, nonpartisan social and education policy research organization dedicated 
to learning what works to improve the well-being of low-income people. Through its research 
and the active communication of its findings, MDRC seeks to enhance the effectiveness of so- 
cial and education policies and programs. 


Founded in 1974 and located in New York; Oakland, California; Washington, DC; and Los 
Angeles, MDRC is best known for mounting rigorous, large-scale, real-world tests of new and 
existing policies and programs. Its projects are a mix of demonstrations (field tests of promising 
new program approaches) and evaluations of ongoing government and community initiatives. 
MDRC’s staff members bring an unusual combination of research and organizational experi- 
ence to their work, providing expertise on the latest in qualitative and quantitative methods and 
on program design, development, implementation, and management. MDRC seeks to learn not 
just whether a program is effective but also how and why the program’s effects occur. In addi- 
tion, it tries to place each project’s findings in the broader context of related research — in order 
to build knowledge about what works across the social and education policy fields. MDRC’s 
findings, lessons, and best practices are shared with a broad audience in the policy and practi- 
tioner community as well as with the general public and the media. 


Over the years, MDRC has brought its unique approach to an ever-growing range of policy are- 
as and target populations. Once known primarily for evaluations of state welfare-to-work pro- 
grams, today MDRC is also studying public school reforms, employment programs for ex- 
prisoners, and programs to help low-income students succeed in college. MDRC’s projects are 
organized into five areas: 

e Promoting Family Well-Being and Children’s Development 

e Improving Public Education 

e Raising Academic Achievement and Persistence in College 

e Supporting Low-Wage Workers and Communities 


e Overcoming Barriers to Employment 


Working in almost every state, all of the nation’s largest cities, and Canada and the United 
Kingdom, MDRC conducts its projects in partnership with national, state, and local govern- 
ments, public school systems, community organizations, and numerous private philanthropies. 


